apply each agent's internet flag to its squid credential
This commit is contained in:
atlas
mara
parent
0b99f17bc3
commit
501445ebc3
8 changed files with 323 additions and 91 deletions
|
|
@ -230,17 +230,33 @@ host. It never decrypts TLS, so it holds no signing key. It caches nothing.
|
|||
store and the next pass mints a new one. HTTP Basic crosses the bridge in
|
||||
clear between an agent and the proxy. See
|
||||
[the credentials reference](../swarm/credentials-reference.md).
|
||||
A name that isn't on the allow-list gets the 403 whether or not the request
|
||||
has a credential; a listed name without one gets a 407.
|
||||
A request without a valid credential gets a 407, once the destination
|
||||
checks below have passed it.
|
||||
|
||||
- **Two lists.** The credential decides the policy. `hive-c0re` writes a
|
||||
second file beside the users file, `services.hyperhive.squid.unrestrictedFile`
|
||||
(`/var/lib/hive-squid/unrestricted`, `hive-core:squid`, `0640`): the names of
|
||||
the users whose agent holds the `unrestricted_internet` flag in the
|
||||
permissions the hive last read from the swarm, one per line, rewritten when
|
||||
it changes. An agent with no record, or a record that never set the flag,
|
||||
holds it, the same default as the
|
||||
`hyperhive.agent.unrestricted_internet` metric. Squid reads the list when it
|
||||
parses its configuration, so `squid-unrestricted-reload.path` reloads squid
|
||||
when the file changes. A missing or empty file lists no one, and squid
|
||||
starts either way. A flag change reaches squid after the hive's next read of
|
||||
the permissions and its next 30-second pass.
|
||||
|
||||
- **Allow-list.** `services.hyperhive.squid.allowedDomains` defaults to the
|
||||
swarm's own service names (`services.hyperhive.swarm.serviceDomains`). Each
|
||||
entry matches that exact name. Everything else gets a 403.
|
||||
- **Unlisted names.** Refused first, without a DNS lookup.
|
||||
entry matches that exact name. A user on the unrestricted list reaches any
|
||||
other name too, on ports 80 and 443. Everyone else gets a 403 for it.
|
||||
- **Unlisted names.** For a user who isn't on the unrestricted list, refused
|
||||
after the destination checks, which look the name up on the host.
|
||||
- **Denied before any allow.** Destinations that resolve to loopback,
|
||||
link-local or the bridge subnet, and IP-literal hosts. Squid resolves names
|
||||
on the host, so without these a listed name pointing at `127.0.0.1` would
|
||||
walk around the bridge→loopback DROP rule. Only `CONNECT` to port 443 and
|
||||
link-local or the bridge subnet, and IP-literal hosts, for every user
|
||||
including the unrestricted ones. Squid resolves names on the host, so
|
||||
without these a name pointing at `127.0.0.1` would walk around the
|
||||
bridge→loopback DROP rule. Only `CONNECT` to port 443 and
|
||||
plain HTTP to ports 80 and 443 pass.
|
||||
- **Metrics.** `squid-exporter` serves `prometheus-squid-exporter` on
|
||||
`127.0.0.1:9301`. The hive collector scrapes it as job `squid` through
|
||||
|
|
@ -254,9 +270,13 @@ host. It never decrypts TLS, so it holds no signing key. It caches nothing.
|
|||
every minute from the bridge IP, logging in as `selfcheck.squid`: `CONNECT`
|
||||
to `services.hyperhive.squid.selfcheck.canary` (default: the forge's name)
|
||||
must answer 200, to `deny-canary.invalid` 403, and to the canary without a
|
||||
credential 407. It writes one `squid-selfcheck ok=… allow_status=…
|
||||
deny_status=… anon_status=… credential=present|absent` line per run and
|
||||
fails the unit when any is wrong. The password is the hive's
|
||||
credential 407. It then logs in as `selfcheck-unrestricted.squid`, which has
|
||||
the same secret and is always on the unrestricted list, and a `CONNECT` to
|
||||
`services.hyperhive.squid.selfcheck.unrestrictedCanary` (default:
|
||||
`example.com`, a name outside the allow-list) must answer 200. It writes one
|
||||
`squid-selfcheck ok=… allow_status=… deny_status=… anon_status=…
|
||||
credential=present|absent unrestricted_canary=… unrestricted_status=…` line
|
||||
per run and fails the unit when any is wrong. The password is the hive's
|
||||
`swarm/hives/<hive>/squid-selfcheck` secret, which
|
||||
`squid-selfcheck-credential` lands at `/run/credstore/hive-squid.selfcheck`.
|
||||
Before `swarm-controller` has minted it, or while `hive-c0re` hasn't
|
||||
|
|
|
|||
|
|
@ -320,7 +320,10 @@ an absent flag included, and a value in the request wins. A `PUT` that leaves
|
|||
the field out and a `DELETE` (`swarmctl agent permissions clear`) also keep the
|
||||
stored value; a `DELETE` on an agent that has one rewrites the record to just
|
||||
that value. Read it with
|
||||
`swarm_queue_client::agent_permissions::unrestricted`. Nothing acts on it yet.
|
||||
`swarm_queue_client::agent_permissions::unrestricted`. The hive reads it for
|
||||
the agent's squid policy: [the egress proxy](../networking/network.md#egress-proxy-squid)
|
||||
lets an unrestricted agent's credential reach any destination, and holds the
|
||||
others to the allow-list. No agent sends a credential yet.
|
||||
|
||||
**How a change applies** —
|
||||
- The agent's harness reads its own key every 10 seconds and builds every
|
||||
|
|
|
|||
|
|
@ -148,16 +148,16 @@ pieces fit, and the reasoning behind them, is in [`credentials.md`](credentials.
|
|||
|
||||
### Hive-level — one of each per hive
|
||||
|
||||
| secret | generated by | lives at |
|
||||
| ---------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
|
||||
| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` |
|
||||
| the cache's public key and pull token, for hive-priv | `swarm-controller` copies them to `swarm/hives/<hive>/attic/{public-key,pull-token}` (see the cache-facts row above) | `hive-priv-cache-facts` (root) reads those two under the hive's own store identity every ten minutes and writes `/run/hyperhive-priv/attic/public-key` and `netrc` (tmpfs, root, each `0600`, directory `0700`); the cache's address comes from `services.hyperhive.swarm.attic.domain` and `cache`, not from the store. A store that holds no facts, or that doesn't answer, leaves the files as they stand, and a hive that never got them has `hive-priv` answer `not available`. |
|
||||
| the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/run/hive-forge-oidc/<id>.secret` on the host (tmpfs), bound into the forge container, handed to `forgejo-sso-source` through `LoadCredential` |
|
||||
| the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/<id>.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` |
|
||||
| this hive's forge read token | `swarm-controller`, into the secret store at `swarm/hives/<hive>/forge/token`, for the `hive-<hive>` account; a pass at start and every five minutes re-mints a missing or stale token | `hive-c0re` reads it under the hive's own store identity, and copies it (account on the first line, token on the second) to `/run/hyperhive/forge-hive-token` (tmpfs, `0600` `hive-core`) for nix's `git`, which `hive-priv` also runs. Each `internal/meta` read and each boot rewrites the copy when it differs; where they differ after a rotation, a build that starts before the next read fails to authenticate, and a retry reads the new token |
|
||||
| squid's users file | `hive-c0re` (`workers::squid_users`), every 30 seconds, from the proxy credentials of its agents and the self-check | `services.hyperhive.squid.usersFile` (`/var/lib/hive-squid/users`, `hive-core:squid`, `0640`): `<user>:<hash>` lines, SHA-512 crypt hashes only, renamed into place when it changed. `systemd-tmpfiles` creates it empty so squid's helper never sees a missing file. A hive whose agents have no credential yet (an older controller) has an empty file: every login fails and the self-check reports `ok=false`; no agent sends a credential, so no traffic changes |
|
||||
| the squid self-check's password | `swarm-controller` mints `swarm/hives/<hive>/squid-selfcheck`; `squid-selfcheck-credential` (root, every 10 minutes) reads it as the hive | `/run/credstore/hive-squid.selfcheck` (tmpfs, root, `0400`), imported by `squid-selfcheck.service` as a systemd credential and handed to `curl` on its stdin. Missing, or different from the users file: the self-check logs `credential=absent` or a 407 and fails, and nothing else breaks |
|
||||
| secret | generated by | lives at |
|
||||
| ---------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
|
||||
| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` |
|
||||
| the cache's public key and pull token, for hive-priv | `swarm-controller` copies them to `swarm/hives/<hive>/attic/{public-key,pull-token}` (see the cache-facts row above) | `hive-priv-cache-facts` (root) reads those two under the hive's own store identity every ten minutes and writes `/run/hyperhive-priv/attic/public-key` and `netrc` (tmpfs, root, each `0600`, directory `0700`); the cache's address comes from `services.hyperhive.swarm.attic.domain` and `cache`, not from the store. A store that holds no facts, or that doesn't answer, leaves the files as they stand, and a hive that never got them has `hive-priv` answer `not available`. |
|
||||
| the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/run/hive-forge-oidc/<id>.secret` on the host (tmpfs), bound into the forge container, handed to `forgejo-sso-source` through `LoadCredential` |
|
||||
| the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/<id>.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` |
|
||||
| this hive's forge read token | `swarm-controller`, into the secret store at `swarm/hives/<hive>/forge/token`, for the `hive-<hive>` account; a pass at start and every five minutes re-mints a missing or stale token | `hive-c0re` reads it under the hive's own store identity, and copies it (account on the first line, token on the second) to `/run/hyperhive/forge-hive-token` (tmpfs, `0600` `hive-core`) for nix's `git`, which `hive-priv` also runs. Each `internal/meta` read and each boot rewrites the copy when it differs; where they differ after a rotation, a build that starts before the next read fails to authenticate, and a retry reads the new token |
|
||||
| squid's users file | `hive-c0re` (`workers::squid_users`), every 30 seconds, from the proxy credentials of its agents and the self-check | `services.hyperhive.squid.usersFile` (`/var/lib/hive-squid/users`, `hive-core:squid`, `0640`): `<user>:<hash>` lines, SHA-512 crypt hashes only, renamed into place when it changed. `unrestricted` beside it lists the users whose agent holds the swarm's `unrestricted_internet` flag, names only; it lists every agent the hive never read as off `systemd-tmpfiles` creates it empty so squid's helper never sees a missing file. A hive whose agents have no credential yet (an older controller) has an empty file: every login fails and the self-check reports `ok=false`; no agent sends a credential, so no traffic changes |
|
||||
| the squid self-check's password | `swarm-controller` mints `swarm/hives/<hive>/squid-selfcheck`; `squid-selfcheck-credential` (root, every 10 minutes) reads it as the hive | `/run/credstore/hive-squid.selfcheck` (tmpfs, root, `0400`), imported by `squid-selfcheck.service` as a systemd credential and handed to `curl` on its stdin. Both self-check users (`selfcheck.squid` and the always-unrestricted `selfcheck-unrestricted.squid`) use it, so whoever holds it has unrestricted egress through this squid Missing, or different from the users file: the self-check logs `credential=absent` or a 407 and fails, and nothing else breaks |
|
||||
|
||||
### Store identities, one per principal
|
||||
|
||||
|
|
|
|||
|
|
@ -568,7 +568,9 @@ the swarm never set the flag, which counts as on. The **restrict** or
|
|||
**allow unrestricted** badge flips it with a `PUT` that carries the stored
|
||||
permission list along with the flag, because a `PUT` without the list resets a
|
||||
stored list to the default. An agent with no stored list sends the flag alone.
|
||||
Nothing acts on the flag yet.
|
||||
The hive's egress proxy applies the flag to each agent's proxy credential (see
|
||||
[the egress proxy](../networking/network.md#egress-proxy-squid)); no agent sends
|
||||
one yet, so the flag changes no traffic today.
|
||||
|
||||
A saved change isn't instant. The hive reads the caps about every 30 seconds,
|
||||
and a running agent takes a changed cap at its next spawn or rebuild, so the
|
||||
|
|
|
|||
|
|
@ -272,8 +272,11 @@ in
|
|||
BAO_CLIENT_CERT = "%d/bao-client.pem";
|
||||
BAO_CLIENT_KEY = "%d/bao-client-key.pem";
|
||||
# Where `workers::squid_users` renders the hashes of this hive's agents' proxy
|
||||
# credentials for squid; it reads them from the store under the identity above.
|
||||
# credentials for squid, and the list of the users whose agent holds the
|
||||
# swarm's `unrestricted_internet` flag; it reads the credentials from the
|
||||
# store under the identity above.
|
||||
HIVE_SQUID_USERS_FILE = config.services.hyperhive.squid.usersFile;
|
||||
HIVE_SQUID_UNRESTRICTED_FILE = config.services.hyperhive.squid.unrestrictedFile;
|
||||
}
|
||||
// lib.optionalAttrs (haveBaoClientIdentity && baoDeploy.serverCaFile != null) {
|
||||
# Absent means the system trust store — right for a deployment with a real
|
||||
|
|
|
|||
|
|
@ -1,9 +1,12 @@
|
|||
# Egress proxy for this hive's agents: squid on the host, bound to the bridge
|
||||
# IP, default-deny on the destination hostname. It filters CONNECT targets and
|
||||
# plain-HTTP hosts and never decrypts, so it holds no signing key. A request
|
||||
# for a listed name must carry a per-agent proxy credential, checked against the
|
||||
# users file hive-c0re renders. No agent sends one yet; the exporter, journal
|
||||
# forwarding and the self-check below are how a deploy shows it works.
|
||||
# must carry a per-agent proxy credential, checked against the users file
|
||||
# hive-c0re renders. A credential's agent either has the swarm's
|
||||
# `unrestricted_internet` flag, and may then reach any destination the deny
|
||||
# rules leave, or reaches only the listed names. No agent sends a credential
|
||||
# yet; the exporter, journal forwarding and the self-check below are how a
|
||||
# deploy shows it works.
|
||||
{
|
||||
pkgs,
|
||||
lib,
|
||||
|
|
@ -24,18 +27,21 @@ let
|
|||
|
||||
# The squid user the self-check logs in as: `swarm_secret_client::proxy::SELFCHECK_USER`.
|
||||
selfcheckUser = "selfcheck.squid";
|
||||
# The same string in `swarm_secret_client::proxy::SELFCHECK_UNRESTRICTED_USER`.
|
||||
selfcheckUnrestrictedUser = "selfcheck-unrestricted.squid";
|
||||
usersDir = builtins.dirOf cfg.usersFile;
|
||||
|
||||
# Evaluated top to bottom, first match wins. The three destination denies
|
||||
# sit before the only allow: squid resolves names on the host, so a name
|
||||
# that answers 127.0.0.1, 169.254.169.254 or a bridge address would
|
||||
# otherwise reach services the bridge firewall keeps agents out of. The
|
||||
# unlisted-name deny comes first because `dstdomain` needs no lookup while
|
||||
# the `dst` ACLs do: an unlisted name is refused at once instead of waiting
|
||||
# on the host resolver, and costs a client no DNS query. The credential
|
||||
# check sits right before the allow, so a refused destination answers 403
|
||||
# to a client with or without a credential, and only a request that would
|
||||
# otherwise be let through gets the 407 challenge.
|
||||
# sit before every allow, `allow unrestricted` included: squid resolves
|
||||
# names on the host, so a name that answers 127.0.0.1, 169.254.169.254 or a
|
||||
# bridge address would otherwise reach services the bridge firewall keeps
|
||||
# agents out of, and an unrestricted agent is still an agent. The IP-literal
|
||||
# denies come first because `dstdomain` on an IP literal does a reverse
|
||||
# lookup. The credential check sits right before the first allow, so only a
|
||||
# request that would otherwise be let through gets the 407 challenge. An
|
||||
# unrestricted user is then let through to any other destination on a safe
|
||||
# port, and every other user to the listed names. The `dst` denies cost an
|
||||
# unlisted name a lookup on the host's resolver before it is refused.
|
||||
squidConf = ''
|
||||
http_port ${netCfg.bridgeIp}:${toString port}
|
||||
pid_filename /run/squid/squid.pid
|
||||
|
|
@ -59,6 +65,10 @@ let
|
|||
acl SSL_ports port 443
|
||||
acl Safe_ports port 80 443
|
||||
acl authed proxy_auth REQUIRED
|
||||
# The users whose agent holds the flag. Squid reads the file when it parses
|
||||
# its configuration, so `squid-unrestricted-reload` reloads squid when
|
||||
# hive-c0re rewrites it. An empty or missing file lists no one.
|
||||
acl unrestricted proxy_auth "${cfg.unrestrictedFile}"
|
||||
|
||||
# Host-originated requests: the self-check and the exporter. A local
|
||||
# connect to the bridge IP takes the bridge IP as its source.
|
||||
|
|
@ -82,11 +92,12 @@ let
|
|||
http_access deny CONNECT !SSL_ports
|
||||
http_access deny ip_v4_host
|
||||
http_access deny ip_v6_host
|
||||
${lib.optionalString (allowed != [ ]) "http_access deny !allowed_dst"}
|
||||
http_access deny to_localhost
|
||||
http_access deny to_linklocal
|
||||
http_access deny to_bridge
|
||||
http_access deny !authed
|
||||
http_access allow unrestricted
|
||||
${lib.optionalString (allowed != [ ]) "http_access deny !allowed_dst"}
|
||||
${lib.optionalString (allowed != [ ]) "http_access allow allowed_dst"}
|
||||
http_access deny all
|
||||
'';
|
||||
|
|
@ -128,30 +139,32 @@ let
|
|||
secret=$(< "$credential_file")
|
||||
fi
|
||||
|
||||
# <host> <auth|anon>: the status squid answers a CONNECT with; 000 when
|
||||
# the proxy is down.
|
||||
# <host> <user|anon>: the status squid answers a CONNECT with; 000 when
|
||||
# the proxy is down. Both users log in with the one secret.
|
||||
connect_status() {
|
||||
{
|
||||
if [ "$2" = auth ] && [ -n "$secret" ]; then
|
||||
printf 'proxy-user = "%s:%s"\n' ${selfcheckUser} "$secret"
|
||||
if [ "$2" != anon ] && [ -n "$secret" ]; then
|
||||
printf 'proxy-user = "%s:%s"\n' "$2" "$secret"
|
||||
fi
|
||||
} | curl -sS -m 10 -K - --interface ${netCfg.bridgeIp} -p -x ${proxyUrl} \
|
||||
-o /dev/null -w '%{http_connect}' "https://$1/" 2>/dev/null || true
|
||||
}
|
||||
|
||||
allow_status=$(connect_status ${cfg.selfcheck.canary} auth)
|
||||
deny_status=$(connect_status ${denyCanary} auth)
|
||||
allow_status=$(connect_status ${cfg.selfcheck.canary} ${selfcheckUser})
|
||||
deny_status=$(connect_status ${denyCanary} ${selfcheckUser})
|
||||
anon_status=$(connect_status ${cfg.selfcheck.canary} anon)
|
||||
unrestricted_status=$(connect_status ${cfg.selfcheck.unrestrictedCanary} ${selfcheckUnrestrictedUser})
|
||||
|
||||
ok=false
|
||||
if [ "$allow_status" = 200 ] && [ "$deny_status" = 403 ] && [ "$anon_status" = 407 ]; then
|
||||
if [ "$allow_status" = 200 ] && [ "$deny_status" = 403 ] && [ "$anon_status" = 407 ] \
|
||||
&& [ "$unrestricted_status" = 200 ]; then
|
||||
ok=true
|
||||
fi
|
||||
credential=absent
|
||||
if [ -n "$secret" ]; then
|
||||
credential=present
|
||||
fi
|
||||
echo "squid-selfcheck ok=$ok allow_canary=${cfg.selfcheck.canary} allow_status=$allow_status deny_canary=${denyCanary} deny_status=$deny_status anon_status=$anon_status credential=$credential"
|
||||
echo "squid-selfcheck ok=$ok allow_canary=${cfg.selfcheck.canary} allow_status=$allow_status deny_canary=${denyCanary} deny_status=$deny_status anon_status=$anon_status credential=$credential unrestricted_canary=${cfg.selfcheck.unrestrictedCanary} unrestricted_status=$unrestricted_status"
|
||||
[ "$ok" = true ]
|
||||
'';
|
||||
};
|
||||
|
|
@ -186,6 +199,23 @@ in
|
|||
'';
|
||||
};
|
||||
|
||||
unrestrictedFile = lib.mkOption {
|
||||
type = lib.types.path;
|
||||
default = "${builtins.dirOf cfg.usersFile}/unrestricted";
|
||||
readOnly = true;
|
||||
defaultText = lib.literalMD "`unrestricted`, in the directory of `usersFile`";
|
||||
description = ''
|
||||
The user names, one per line, that squid lets reach any destination the
|
||||
deny rules leave open: the users of the agents that hold the swarm's
|
||||
`unrestricted_internet` flag (set unless the swarm records it off), and
|
||||
the self-check's unrestricted user. hive-c0re renders it beside
|
||||
`usersFile`, and squid is reloaded when it changes. Every other user
|
||||
reaches only `allowedDomains`. An empty file, which is what exists
|
||||
until hive-c0re writes one, makes every user a restricted one.
|
||||
Owned by `hive-core` and readable by the `squid` group.
|
||||
'';
|
||||
};
|
||||
|
||||
selfcheck.credentialName = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
default = "hive-squid.selfcheck";
|
||||
|
|
@ -198,6 +228,17 @@ in
|
|||
'';
|
||||
};
|
||||
|
||||
selfcheck.unrestrictedCanary = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
default = "example.com";
|
||||
description = ''
|
||||
Hostname outside `allowedDomains` that the self-check CONNECTs to on
|
||||
port 443, as its unrestricted user, and expects squid to let through.
|
||||
Squid resolves it with this host's resolver and connects to it, so a
|
||||
host with no route to it reports `ok=false`.
|
||||
'';
|
||||
};
|
||||
|
||||
selfcheck.canary = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
default = hyperhiveCfg.swarm.forge.domain;
|
||||
|
|
@ -230,11 +271,43 @@ in
|
|||
};
|
||||
|
||||
# The directory is setgid so the file hive-c0re renames into it is the
|
||||
# squid group's. The file exists before squid starts, empty: the helper
|
||||
# exits on a missing file and squid then rejects every login.
|
||||
# squid group's. The users file exists before squid starts, empty: the
|
||||
# helper exits on a missing file and squid then rejects every login. The
|
||||
# unrestricted list is created the same way, though squid treats a missing
|
||||
# one as empty.
|
||||
systemd.tmpfiles.rules = [
|
||||
"d ${usersDir} 2750 hive-core squid - -"
|
||||
"f ${cfg.usersFile} 0640 hive-core squid - -"
|
||||
"f ${cfg.unrestrictedFile} 0640 hive-core squid - -"
|
||||
];
|
||||
|
||||
# Squid reads the unrestricted list when it parses its configuration, not
|
||||
# when it changes. hive-c0re writes the list only when its contents change,
|
||||
# so each change is one reload. Nothing happens to a squid that is not
|
||||
# running; it reads the list when it starts.
|
||||
systemd.paths.squid-unrestricted-reload = {
|
||||
description = "Watch squid's unrestricted-users list";
|
||||
wantedBy = [ "paths.target" ];
|
||||
pathConfig.PathChanged = cfg.unrestrictedFile;
|
||||
};
|
||||
|
||||
systemd.services.squid-unrestricted-reload = {
|
||||
description = "Reload squid to read its unrestricted-users list";
|
||||
serviceConfig = {
|
||||
Type = "oneshot";
|
||||
ExecStart = "${config.systemd.package}/bin/systemctl try-reload-or-restart squid.service";
|
||||
};
|
||||
};
|
||||
|
||||
assertions = [
|
||||
{
|
||||
assertion = !(builtins.elem cfg.selfcheck.unrestrictedCanary allowed);
|
||||
message = ''
|
||||
services.hyperhive.squid.selfcheck.unrestrictedCanary must be a name outside
|
||||
allowedDomains, or the self-check cannot tell an unrestricted user from a
|
||||
restricted one.
|
||||
'';
|
||||
}
|
||||
];
|
||||
|
||||
systemd.services.squid = {
|
||||
|
|
|
|||
|
|
@ -485,38 +485,80 @@ let
|
|||
&& !(lib.hasInfix "localnet" conf);
|
||||
}
|
||||
{
|
||||
# First match wins, so a destination deny after the allow is dead, and
|
||||
# a credential check after it never challenges anyone. The two IP-literal
|
||||
# denies stay ahead of the unlisted-name deny: `dstdomain` on an IP
|
||||
# First match wins, so a destination deny after an allow is dead, and a
|
||||
# credential check after one never challenges anyone. `allow unrestricted`
|
||||
# is an allow: above the three destination denies an unrestricted user
|
||||
# reaches loopback, link-local and the bridge through the proxy. The two
|
||||
# IP-literal denies stay ahead of every allow: `dstdomain` on an IP
|
||||
# literal does a reverse lookup.
|
||||
name = "squid refuses unlisted names, then loopback, link-local and bridge destinations, then a missing credential, before any allow";
|
||||
name = "squid refuses loopback, link-local and bridge destinations and IP literals, then a missing credential, before any allow";
|
||||
ok =
|
||||
let
|
||||
lines = lib.splitString "\n" hiveOtel.services.squid.configText;
|
||||
at = l: lib.lists.findFirstIndex (x: x == l) null lines;
|
||||
allow = at "http_access allow allowed_dst";
|
||||
unrestricted = at "http_access allow unrestricted";
|
||||
authed = at "http_access deny !authed";
|
||||
firstAllow = if unrestricted < allow then unrestricted else allow;
|
||||
in
|
||||
allow != null
|
||||
&& unrestricted != null
|
||||
&& authed != null
|
||||
&& at "http_access deny to_localhost" != null
|
||||
&& at "http_access deny to_localhost" < allow
|
||||
&& at "http_access deny to_linklocal" != null
|
||||
&& at "http_access deny to_linklocal" < allow
|
||||
&& at "http_access deny to_bridge" != null
|
||||
&& at "http_access deny to_bridge" < allow
|
||||
&& at "http_access deny ip_v4_host" != null
|
||||
&& at "http_access deny ip_v6_host" != null
|
||||
&& lib.all (l: at l < firstAllow) [
|
||||
"http_access deny to_localhost"
|
||||
"http_access deny to_linklocal"
|
||||
"http_access deny to_bridge"
|
||||
"http_access deny ip_v4_host"
|
||||
"http_access deny ip_v6_host"
|
||||
]
|
||||
&& at "http_access deny to_localhost" < authed
|
||||
&& at "http_access deny to_linklocal" < authed
|
||||
&& at "http_access deny to_bridge" < authed
|
||||
&& authed < allow
|
||||
&& at "http_access deny ip_v4_host" != null
|
||||
&& at "http_access deny ip_v4_host" < at "http_access deny !allowed_dst"
|
||||
&& at "http_access deny ip_v6_host" < at "http_access deny !allowed_dst"
|
||||
&& at "http_access deny !allowed_dst" != null
|
||||
&& at "http_access deny !allowed_dst" < at "http_access deny to_localhost"
|
||||
&& authed < firstAllow
|
||||
&& builtins.elem "cache deny all" lines
|
||||
&& lib.last (lib.filter (l: l != "") lines) == "http_access deny all";
|
||||
}
|
||||
{
|
||||
# An unrestricted user is let through before the unlisted-name deny and
|
||||
# everyone else is held to the listed names after it. Control: both are
|
||||
# there, so the ordering below is not a lookup of nothing.
|
||||
name = "squid lets an unrestricted user through before the allow-list rules that bind everyone else";
|
||||
ok =
|
||||
let
|
||||
conf = hiveOtel.services.squid.configText;
|
||||
lines = lib.splitString "\n" conf;
|
||||
at = l: lib.lists.findFirstIndex (x: x == l) null lines;
|
||||
list = hiveOtel.services.hyperhive.squid.unrestrictedFile;
|
||||
in
|
||||
at "http_access allow unrestricted" != null
|
||||
&& at "http_access deny !allowed_dst" != null
|
||||
&& at "http_access allow unrestricted" < at "http_access deny !allowed_dst"
|
||||
&& at "http_access deny !allowed_dst" < at "http_access allow allowed_dst"
|
||||
&& builtins.elem ''acl unrestricted proxy_auth "${list}"'' lines
|
||||
&& list == "/var/lib/hive-squid/unrestricted";
|
||||
}
|
||||
{
|
||||
# The list is read when squid parses its configuration, so a change to
|
||||
# the file reloads squid. Control: the watched path is the list's own.
|
||||
name = "squid is reloaded when its unrestricted-users list changes";
|
||||
ok =
|
||||
let
|
||||
list = hiveOtel.services.hyperhive.squid.unrestrictedFile;
|
||||
rules = hiveOtel.systemd.tmpfiles.rules;
|
||||
path = hiveOtel.systemd.paths.squid-unrestricted-reload;
|
||||
svc = hiveOtel.systemd.services.squid-unrestricted-reload.serviceConfig;
|
||||
in
|
||||
path.pathConfig.PathChanged == list
|
||||
&& builtins.elem "paths.target" path.wantedBy
|
||||
&& lib.hasSuffix "systemctl try-reload-or-restart squid.service" svc.ExecStart
|
||||
&& builtins.elem "f ${list} 0640 hive-core squid - -" rules
|
||||
&& hiveOtelAllLocal.systemd.services.hive-c0re.environment.HIVE_SQUID_UNRESTRICTED_FILE == list;
|
||||
}
|
||||
{
|
||||
# The helper is the NCSA one, on the file hive-c0re renders. The file
|
||||
# exists, empty, before squid starts (the helper exits on a missing one),
|
||||
|
|
|
|||
|
|
@ -112,13 +112,18 @@ pkgs.testers.runNixOSTest {
|
|||
];
|
||||
# Names the proxy resolves, each a different destination class.
|
||||
networking.hosts = {
|
||||
"192.168.2.2" = [ "forge.t.local" ];
|
||||
# `other.t.local` resolves, is reachable and is on no allow-list.
|
||||
"192.168.2.2" = [
|
||||
"forge.t.local"
|
||||
"other.t.local"
|
||||
];
|
||||
"127.0.0.1" = [ "loop.t.local" ];
|
||||
${bridgeIp} = [ "bridge.t.local" ];
|
||||
"169.254.169.254" = [ "meta.t.local" ];
|
||||
};
|
||||
# Renders the users file the way hive-c0re does.
|
||||
environment.systemPackages = [ pkgs.mkpasswd ];
|
||||
services.hyperhive.squid.selfcheck.unrestrictedCanary = "other.t.local";
|
||||
services.hyperhive.squid.allowedDomains = [
|
||||
"forge.t.local"
|
||||
"loop.t.local"
|
||||
|
|
@ -157,14 +162,24 @@ pkgs.testers.runNixOSTest {
|
|||
testScript = ''
|
||||
import re
|
||||
import secrets
|
||||
import time
|
||||
|
||||
# Drawn fresh each run. The users file holds only their hashes, as
|
||||
# hive-c0re renders it.
|
||||
PW = {u: secrets.token_urlsafe(24) for u in ("atlas", "selfcheck.squid", "ruth")}
|
||||
# hive-c0re renders it. `atlas` is a restricted agent and `uma` an
|
||||
# unrestricted one; the two self-check users share one secret.
|
||||
PW = {
|
||||
u: secrets.token_urlsafe(24)
|
||||
for u in ("atlas", "uma", "selfcheck.squid", "ruth")
|
||||
}
|
||||
PW["selfcheck-unrestricted.squid"] = PW["selfcheck.squid"]
|
||||
ATLAS = "atlas:" + PW["atlas"]
|
||||
UMA = "uma:" + PW["uma"]
|
||||
SELFCHECK = "selfcheck.squid:" + PW["selfcheck.squid"]
|
||||
SELFCHECK_U = "selfcheck-unrestricted.squid:" + PW["selfcheck.squid"]
|
||||
RUTH = "ruth:" + PW["ruth"]
|
||||
USERS = "/var/lib/hive-squid/users"
|
||||
LIST = "/var/lib/hive-squid/unrestricted"
|
||||
UNRESTRICTED = ["uma", "selfcheck-unrestricted.squid"]
|
||||
|
||||
start_all()
|
||||
proxy.wait_for_unit("squid.service")
|
||||
|
|
@ -194,17 +209,42 @@ pkgs.testers.runNixOSTest {
|
|||
f"&& chmod 640 {USERS}.new && mv {USERS}.new {USERS} && sleep 1"
|
||||
)
|
||||
|
||||
with subtest("squid starts on the empty users file and refuses every login"):
|
||||
def write_list(*names):
|
||||
# What hive-c0re does for the list. Squid reads it on a reload, which
|
||||
# squid-unrestricted-reload does when the file changes.
|
||||
body = "printf '%s\\n' " + " ".join(names) if names else ":"
|
||||
proxy.succeed(
|
||||
f"({body}) > {LIST}.new && chown hive-core:squid {LIST}.new "
|
||||
f"&& chmod 640 {LIST}.new && mv {LIST}.new {LIST}"
|
||||
)
|
||||
|
||||
def wait_status(host, cred, want):
|
||||
# The reload that follows a list write is asynchronous.
|
||||
got = None
|
||||
for _ in range(30):
|
||||
got = connect(host, cred)
|
||||
if got == want:
|
||||
return
|
||||
time.sleep(1)
|
||||
raise AssertionError(f"{host} as {cred.split(':')[0]}: wanted {want}, last got {got}")
|
||||
|
||||
with subtest("squid starts on the empty users file and list and refuses every login"):
|
||||
assert proxy.succeed(f"stat -c '%U:%G %a' {USERS}").strip() == "hive-core:squid 640"
|
||||
assert proxy.succeed(f"stat -c '%U:%G %a' {LIST}").strip() == "hive-core:squid 640"
|
||||
assert proxy.succeed("stat -c %a /var/lib/hive-squid").strip() == "2750"
|
||||
assert proxy.succeed(f"stat -c %s {USERS}").strip() == "0"
|
||||
assert proxy.succeed(f"stat -c %s {LIST}").strip() == "0"
|
||||
assert connect("forge.t.local", ATLAS) == "407", "no such user"
|
||||
assert connect("deny-canary.invalid", ATLAS) == "403", "an unlisted name needs no login"
|
||||
assert connect("other.t.local", UMA) == "407", "no such user, unlisted name"
|
||||
|
||||
with subtest("the users file holds hashes, never a password"):
|
||||
write_users(ATLAS, SELFCHECK)
|
||||
write_users(ATLAS, UMA, SELFCHECK, SELFCHECK_U)
|
||||
users = proxy.succeed(f"cat {USERS}")
|
||||
assert re.fullmatch(r"(atlas|selfcheck\.squid):\$6\$\S+\n(atlas|selfcheck\.squid):\$6\$\S+\n", users), users
|
||||
rows = [l.split(":", 1) for l in users.splitlines()]
|
||||
assert sorted(r[0] for r in rows) == sorted(
|
||||
["atlas", "uma", "selfcheck.squid", "selfcheck-unrestricted.squid"]
|
||||
), users
|
||||
assert all(re.fullmatch(r"\$6\$\S+", r[1]) for r in rows), users
|
||||
assert not any(p in users for p in PW.values()), users
|
||||
|
||||
with subtest("an allowed name is tunnelled with a credential and challenged without one"):
|
||||
|
|
@ -215,22 +255,31 @@ pkgs.testers.runNixOSTest {
|
|||
# DES would accept the first eight characters; the hash must not.
|
||||
assert connect("forge.t.local", "atlas:" + PW["atlas"][:8]) == "407", "a prefix of the password"
|
||||
|
||||
with subtest("an unlisted name is refused with or without a credential"):
|
||||
assert connect("deny-canary.invalid", ATLAS) == "403", "deny canary"
|
||||
assert connect("deny-canary.invalid") == "403", "deny canary, anonymous"
|
||||
with subtest("with an empty list every user is restricted"):
|
||||
assert connect("other.t.local", UMA) == "403", "uma, not listed yet"
|
||||
assert connect("other.t.local", ATLAS) == "403"
|
||||
assert connect("forge.t.local", UMA) == "200", "control: uma still reaches a listed name"
|
||||
|
||||
with subtest("a name resolving to a host-local destination is refused"):
|
||||
assert connect("loop.t.local", ATLAS) == "403", "loopback"
|
||||
assert connect("bridge.t.local", ATLAS) == "403", "bridge address"
|
||||
assert connect("meta.t.local", ATLAS) == "403", "link-local"
|
||||
with subtest("a listed user reaches an unlisted name and the others are refused"):
|
||||
write_list(*UNRESTRICTED)
|
||||
wait_status("other.t.local", UMA, "200")
|
||||
assert connect("other.t.local", ATLAS) == "403", "restricted"
|
||||
assert connect("forge.t.local", ATLAS) == "200", "restricted, listed name"
|
||||
assert connect("forge.t.local", UMA) == "200", "unrestricted, listed name"
|
||||
assert connect("other.t.local") == "407", "no credential"
|
||||
assert connect("other.t.local", "uma:" + secrets.token_urlsafe(24)) == "407", "wrong password"
|
||||
assert connect("deny-canary.invalid", ATLAS) == "403", "restricted, a name that does not resolve"
|
||||
|
||||
with subtest("IP literals are refused even when the address is reachable"):
|
||||
assert connect("127.0.0.1", ATLAS) == "403"
|
||||
assert connect("192.168.2.2", ATLAS) == "403"
|
||||
|
||||
with subtest("only a port-443 CONNECT is allowed"):
|
||||
assert connect("forge.t.local:22", ATLAS) == "403"
|
||||
assert connect("forge.t.local:80", ATLAS) == "403"
|
||||
with subtest("neither kind of user reaches a host-local destination or another port"):
|
||||
for cred in (ATLAS, UMA):
|
||||
who = cred.split(":")[0]
|
||||
assert connect("loop.t.local", cred) == "403", f"{who}: loopback"
|
||||
assert connect("bridge.t.local", cred) == "403", f"{who}: bridge address"
|
||||
assert connect("meta.t.local", cred) == "403", f"{who}: link-local"
|
||||
assert connect("127.0.0.1", cred) == "403", f"{who}: loopback literal"
|
||||
assert connect("192.168.2.2", cred) == "403", f"{who}: IP literal"
|
||||
assert connect("other.t.local:22", cred) == "403", f"{who}: port 22"
|
||||
assert connect("other.t.local:80", cred) == "403", f"{who}: CONNECT to port 80"
|
||||
|
||||
with subtest("an agent cannot read cachemgr"):
|
||||
agent.succeed(
|
||||
|
|
@ -247,32 +296,50 @@ pkgs.testers.runNixOSTest {
|
|||
"journalctl -u squid-selfcheck.service -o cat | "
|
||||
"grep -F 'squid-selfcheck ok=true allow_canary=forge.t.local "
|
||||
"allow_status=200 deny_canary=deny-canary.invalid deny_status=403 "
|
||||
"anon_status=407 credential=present'"
|
||||
"anon_status=407 credential=present "
|
||||
"unrestricted_canary=other.t.local unrestricted_status=200'"
|
||||
)
|
||||
|
||||
with subtest("the self-check fails, and says why, when the unrestricted user is not listed"):
|
||||
write_list()
|
||||
wait_status("other.t.local", UMA, "403")
|
||||
proxy.fail("systemctl start squid-selfcheck.service")
|
||||
proxy.succeed(
|
||||
"journalctl -u squid-selfcheck.service -o cat | "
|
||||
"grep -F 'squid-selfcheck ok=false allow_canary=forge.t.local "
|
||||
"allow_status=200 deny_canary=deny-canary.invalid deny_status=403 "
|
||||
"anon_status=407 credential=present "
|
||||
"unrestricted_canary=other.t.local unrestricted_status=403'"
|
||||
)
|
||||
write_list(*UNRESTRICTED)
|
||||
wait_status("other.t.local", UMA, "200")
|
||||
|
||||
with subtest("the self-check fails, and says why, without a credential or with the wrong one"):
|
||||
proxy.succeed("mv /run/credstore/hive-squid.selfcheck /run/credstore/kept")
|
||||
proxy.fail("systemctl start squid-selfcheck.service")
|
||||
proxy.succeed(
|
||||
"journalctl -u squid-selfcheck.service -o cat | "
|
||||
"grep -F 'squid-selfcheck ok=false allow_canary=forge.t.local "
|
||||
"allow_status=407 deny_canary=deny-canary.invalid deny_status=403 "
|
||||
"anon_status=407 credential=absent'"
|
||||
"allow_status=407 deny_canary=deny-canary.invalid deny_status=407 "
|
||||
"anon_status=407 credential=absent "
|
||||
"unrestricted_canary=other.t.local unrestricted_status=407'"
|
||||
)
|
||||
proxy.succeed(f"printf '%s\\n' {secrets.token_urlsafe(24)} > /run/credstore/hive-squid.selfcheck")
|
||||
proxy.fail("systemctl start squid-selfcheck.service")
|
||||
proxy.succeed(
|
||||
"journalctl -u squid-selfcheck.service -o cat | "
|
||||
"grep -F 'allow_status=407 deny_canary=deny-canary.invalid deny_status=403 "
|
||||
"anon_status=407 credential=present'"
|
||||
"grep -F 'allow_status=407 deny_canary=deny-canary.invalid deny_status=407 "
|
||||
"anon_status=407 credential=present "
|
||||
"unrestricted_canary=other.t.local unrestricted_status=407'"
|
||||
)
|
||||
proxy.succeed("mv -f /run/credstore/kept /run/credstore/hive-squid.selfcheck")
|
||||
|
||||
with subtest("squid's access log reaches the journal under squid.service, with the user"):
|
||||
journal = proxy.succeed("journalctl -u squid.service -o cat")
|
||||
assert re.search(r"192\.168\.1\.3 TCP_TUNNEL/200 .* CONNECT forge\.t\.local:443 atlas ", journal), journal
|
||||
assert re.search(r"192\.168\.1\.3 TCP_TUNNEL/200 .* CONNECT other\.t\.local:443 uma ", journal), journal
|
||||
assert re.search(r"192\.168\.1\.3 TCP_DENIED/407 .* CONNECT forge\.t\.local:443 - ", journal), journal
|
||||
assert re.search(r"192\.168\.1\.3 TCP_DENIED/403 .* CONNECT deny-canary\.invalid:443 ", journal), journal
|
||||
assert re.search(r"192\.168\.1\.3 TCP_DENIED/403 .* CONNECT other\.t\.local:443 atlas ", journal), journal
|
||||
assert re.search(r"192\.168\.1\.3 TCP_DENIED/403 .* CONNECT loop\.t\.local:443 ", journal), journal
|
||||
# Exporter scrapes are cachemgr requests and stay out of the log.
|
||||
assert "squid-internal-mgr" not in journal, journal
|
||||
|
|
@ -287,15 +354,36 @@ pkgs.testers.runNixOSTest {
|
|||
assert m and float(m.group(1)) >= 1, metrics
|
||||
proxy.fail("curl -sf -m 5 http://${bridgeIp}:9301/metrics")
|
||||
|
||||
with subtest("squid survives a reload"):
|
||||
with subtest("squid survives a reload and keeps the list"):
|
||||
proxy.succeed("systemctl reload squid.service")
|
||||
proxy.wait_for_unit("squid.service")
|
||||
assert connect("forge.t.local", ATLAS) == "200"
|
||||
assert connect("other.t.local", UMA) == "200"
|
||||
assert connect("other.t.local", ATLAS) == "403"
|
||||
|
||||
with subtest("a user added while squid runs can log in, with no reload"):
|
||||
write_users(ATLAS, SELFCHECK, RUTH)
|
||||
with subtest("a user added while squid runs can log in, with no reload, and is restricted"):
|
||||
write_users(ATLAS, UMA, SELFCHECK, SELFCHECK_U, RUTH)
|
||||
assert connect("forge.t.local", RUTH) == "200"
|
||||
assert connect("forge.t.local", ATLAS) == "200"
|
||||
assert connect("other.t.local", RUTH) == "403"
|
||||
|
||||
with subtest("a user the list gains is unrestricted after the reload and loses it again"):
|
||||
write_list(*UNRESTRICTED, "ruth")
|
||||
wait_status("other.t.local", RUTH, "200")
|
||||
assert connect("loop.t.local", RUTH) == "403", "still no loopback"
|
||||
write_list(*UNRESTRICTED)
|
||||
wait_status("other.t.local", RUTH, "403")
|
||||
assert connect("other.t.local", UMA) == "200", "control: uma is still listed"
|
||||
|
||||
with subtest("squid starts on a missing list, treats every user as restricted, and recovers when it appears"):
|
||||
proxy.succeed(f"rm {LIST}")
|
||||
proxy.succeed("systemctl restart squid.service")
|
||||
proxy.wait_for_unit("squid.service")
|
||||
proxy.wait_for_open_port(3128, "${bridgeIp}")
|
||||
assert connect("forge.t.local", UMA) == "200", "logins work"
|
||||
assert connect("other.t.local", UMA) == "403", "no list, no unrestricted user"
|
||||
write_list(*UNRESTRICTED)
|
||||
wait_status("other.t.local", UMA, "200")
|
||||
|
||||
with subtest("squid starts on a missing users file, refuses logins, and recovers when it appears"):
|
||||
proxy.succeed(f"rm {USERS}")
|
||||
|
|
@ -303,8 +391,9 @@ pkgs.testers.runNixOSTest {
|
|||
proxy.wait_for_unit("squid.service")
|
||||
proxy.wait_for_open_port(3128, "${bridgeIp}")
|
||||
assert connect("forge.t.local", ATLAS) == "407", "no file, no login"
|
||||
assert connect("deny-canary.invalid", ATLAS) == "403", "an unlisted name still answers"
|
||||
write_users(ATLAS, SELFCHECK)
|
||||
assert connect("other.t.local", UMA) == "407", "no file, no login, even when listed"
|
||||
write_users(ATLAS, UMA, SELFCHECK, SELFCHECK_U)
|
||||
assert connect("forge.t.local", ATLAS) == "200", "the file appeared"
|
||||
assert connect("other.t.local", UMA) == "200", "listed, and now logged in"
|
||||
'';
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in a new issue