vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

apply each agent's internet flag to its squid credential

Browse source
This commit is contained in:
atlas 2026-10-28 14:14:30 +01:00 • committed by mara
commit 501445ebc3
8 changed files with 323 additions and 91 deletions
Download patch file Download diff file Expand all files Collapse all files

40
docs/networking/network.md
View file

@ -230,17 +230,33 @@ host. It never decrypts TLS, so it holds no signing key. It caches nothing.
store and the next pass mints a new one. HTTP Basic crosses the bridge in store and the next pass mints a new one. HTTP Basic crosses the bridge in
clear between an agent and the proxy. See clear between an agent and the proxy. See
[the credentials reference](../swarm/credentials-reference.md). [the credentials reference](../swarm/credentials-reference.md).
A name that isn't on the allow-list gets the 403 whether or not the request A request without a valid credential gets a 407, once the destination
has a credential; a listed name without one gets a 407. checks below have passed it.
- **Two lists.** The credential decides the policy. `hive-c0re` writes a
second file beside the users file, `services.hyperhive.squid.unrestrictedFile`
(`/var/lib/hive-squid/unrestricted`, `hive-core:squid`, `0640`): the names of
the users whose agent holds the `unrestricted_internet` flag in the
permissions the hive last read from the swarm, one per line, rewritten when
it changes. An agent with no record, or a record that never set the flag,
holds it, the same default as the
`hyperhive.agent.unrestricted_internet` metric. Squid reads the list when it
parses its configuration, so `squid-unrestricted-reload.path` reloads squid
when the file changes. A missing or empty file lists no one, and squid
starts either way. A flag change reaches squid after the hive's next read of
the permissions and its next 30-second pass.
- **Allow-list.** `services.hyperhive.squid.allowedDomains` defaults to the - **Allow-list.** `services.hyperhive.squid.allowedDomains` defaults to the
swarm's own service names (`services.hyperhive.swarm.serviceDomains`). Each swarm's own service names (`services.hyperhive.swarm.serviceDomains`). Each
entry matches that exact name. Everything else gets a 403. entry matches that exact name. A user on the unrestricted list reaches any
- **Unlisted names.** Refused first, without a DNS lookup. other name too, on ports 80 and 443. Everyone else gets a 403 for it.
- **Unlisted names.** For a user who isn't on the unrestricted list, refused
after the destination checks, which look the name up on the host.
- **Denied before any allow.** Destinations that resolve to loopback, - **Denied before any allow.** Destinations that resolve to loopback,
link-local or the bridge subnet, and IP-literal hosts. Squid resolves names link-local or the bridge subnet, and IP-literal hosts, for every user
on the host, so without these a listed name pointing at `127.0.0.1` would including the unrestricted ones. Squid resolves names on the host, so
walk around the bridge→loopback DROP rule. Only `CONNECT` to port 443 and without these a name pointing at `127.0.0.1` would walk around the
bridge→loopback DROP rule. Only `CONNECT` to port 443 and
plain HTTP to ports 80 and 443 pass. plain HTTP to ports 80 and 443 pass.
- **Metrics.** `squid-exporter` serves `prometheus-squid-exporter` on - **Metrics.** `squid-exporter` serves `prometheus-squid-exporter` on
`127.0.0.1:9301`. The hive collector scrapes it as job `squid` through `127.0.0.1:9301`. The hive collector scrapes it as job `squid` through
@ -254,9 +270,13 @@ host. It never decrypts TLS, so it holds no signing key. It caches nothing.
every minute from the bridge IP, logging in as `selfcheck.squid`: `CONNECT` every minute from the bridge IP, logging in as `selfcheck.squid`: `CONNECT`
to `services.hyperhive.squid.selfcheck.canary` (default: the forge's name) to `services.hyperhive.squid.selfcheck.canary` (default: the forge's name)
must answer 200, to `deny-canary.invalid` 403, and to the canary without a must answer 200, to `deny-canary.invalid` 403, and to the canary without a
credential 407. It writes one `squid-selfcheck ok=… allow_status=… credential 407. It then logs in as `selfcheck-unrestricted.squid`, which has
deny_status=… anon_status=… credential=present|absent` line per run and the same secret and is always on the unrestricted list, and a `CONNECT` to
fails the unit when any is wrong. The password is the hive's `services.hyperhive.squid.selfcheck.unrestrictedCanary` (default:
`example.com`, a name outside the allow-list) must answer 200. It writes one
`squid-selfcheck ok=… allow_status=… deny_status=… anon_status=…
credential=present|absent unrestricted_canary=… unrestricted_status=…` line
per run and fails the unit when any is wrong. The password is the hive's
`swarm/hives/<hive>/squid-selfcheck` secret, which `swarm/hives/<hive>/squid-selfcheck` secret, which
`squid-selfcheck-credential` lands at `/run/credstore/hive-squid.selfcheck`. `squid-selfcheck-credential` lands at `/run/credstore/hive-squid.selfcheck`.
Before `swarm-controller` has minted it, or while `hive-c0re` hasn't Before `swarm-controller` has minted it, or while `hive-c0re` hasn't

5
docs/process/conventions.md
View file

@ -320,7 +320,10 @@ an absent flag included, and a value in the request wins. A `PUT` that leaves
the field out and a `DELETE` (`swarmctl agent permissions clear`) also keep the the field out and a `DELETE` (`swarmctl agent permissions clear`) also keep the
stored value; a `DELETE` on an agent that has one rewrites the record to just stored value; a `DELETE` on an agent that has one rewrites the record to just
that value. Read it with that value. Read it with
`swarm_queue_client::agent_permissions::unrestricted`. Nothing acts on it yet. `swarm_queue_client::agent_permissions::unrestricted`. The hive reads it for
the agent's squid policy: [the egress proxy](../networking/network.md#egress-proxy-squid)
lets an unrestricted agent's credential reach any destination, and holds the
others to the allow-list. No agent sends a credential yet.
**How a change applies** — **How a change applies** —
- The agent's harness reads its own key every 10 seconds and builds every - The agent's harness reads its own key every 10 seconds and builds every

20
docs/swarm/credentials-reference.md
View file

@ -148,16 +148,16 @@ pieces fit, and the reasoning behind them, is in [`credentials.md`](credentials.
### Hive-level — one of each per hive ### Hive-level — one of each per hive
| secret | generated by | lives at | | secret | generated by | lives at |
| ---------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | ---------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `<deploy.hive-controller.tls.stateDir>/<name>.pem` | | hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` | | the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` |
| the cache's public key and pull token, for hive-priv | `swarm-controller` copies them to `swarm/hives/<hive>/attic/{public-key,pull-token}` (see the cache-facts row above) | `hive-priv-cache-facts` (root) reads those two under the hive's own store identity every ten minutes and writes `/run/hyperhive-priv/attic/public-key` and `netrc` (tmpfs, root, each `0600`, directory `0700`); the cache's address comes from `services.hyperhive.swarm.attic.domain` and `cache`, not from the store. A store that holds no facts, or that doesn't answer, leaves the files as they stand, and a hive that never got them has `hive-priv` answer `not available`. | | the cache's public key and pull token, for hive-priv | `swarm-controller` copies them to `swarm/hives/<hive>/attic/{public-key,pull-token}` (see the cache-facts row above) | `hive-priv-cache-facts` (root) reads those two under the hive's own store identity every ten minutes and writes `/run/hyperhive-priv/attic/public-key` and `netrc` (tmpfs, root, each `0600`, directory `0700`); the cache's address comes from `services.hyperhive.swarm.attic.domain` and `cache`, not from the store. A store that holds no facts, or that doesn't answer, leaves the files as they stand, and a hive that never got them has `hive-priv` answer `not available`. |
| the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/run/hive-forge-oidc/<id>.secret` on the host (tmpfs), bound into the forge container, handed to `forgejo-sso-source` through `LoadCredential` | | the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/run/hive-forge-oidc/<id>.secret` on the host (tmpfs), bound into the forge container, handed to `forgejo-sso-source` through `LoadCredential` |
| the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/<id>.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` | | the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/<id>.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` |
| this hive's forge read token | `swarm-controller`, into the secret store at `swarm/hives/<hive>/forge/token`, for the `hive-<hive>` account; a pass at start and every five minutes re-mints a missing or stale token | `hive-c0re` reads it under the hive's own store identity, and copies it (account on the first line, token on the second) to `/run/hyperhive/forge-hive-token` (tmpfs, `0600` `hive-core`) for nix's `git`, which `hive-priv` also runs. Each `internal/meta` read and each boot rewrites the copy when it differs; where they differ after a rotation, a build that starts before the next read fails to authenticate, and a retry reads the new token | | this hive's forge read token | `swarm-controller`, into the secret store at `swarm/hives/<hive>/forge/token`, for the `hive-<hive>` account; a pass at start and every five minutes re-mints a missing or stale token | `hive-c0re` reads it under the hive's own store identity, and copies it (account on the first line, token on the second) to `/run/hyperhive/forge-hive-token` (tmpfs, `0600` `hive-core`) for nix's `git`, which `hive-priv` also runs. Each `internal/meta` read and each boot rewrites the copy when it differs; where they differ after a rotation, a build that starts before the next read fails to authenticate, and a retry reads the new token |
| squid's users file | `hive-c0re` (`workers::squid_users`), every 30 seconds, from the proxy credentials of its agents and the self-check | `services.hyperhive.squid.usersFile` (`/var/lib/hive-squid/users`, `hive-core:squid`, `0640`): `<user>:<hash>` lines, SHA-512 crypt hashes only, renamed into place when it changed. `systemd-tmpfiles` creates it empty so squid's helper never sees a missing file. A hive whose agents have no credential yet (an older controller) has an empty file: every login fails and the self-check reports `ok=false`; no agent sends a credential, so no traffic changes | | squid's users file | `hive-c0re` (`workers::squid_users`), every 30 seconds, from the proxy credentials of its agents and the self-check | `services.hyperhive.squid.usersFile` (`/var/lib/hive-squid/users`, `hive-core:squid`, `0640`): `<user>:<hash>` lines, SHA-512 crypt hashes only, renamed into place when it changed. `unrestricted` beside it lists the users whose agent holds the swarm's `unrestricted_internet` flag, names only; it lists every agent the hive never read as off `systemd-tmpfiles` creates it empty so squid's helper never sees a missing file. A hive whose agents have no credential yet (an older controller) has an empty file: every login fails and the self-check reports `ok=false`; no agent sends a credential, so no traffic changes |
| the squid self-check's password | `swarm-controller` mints `swarm/hives/<hive>/squid-selfcheck`; `squid-selfcheck-credential` (root, every 10 minutes) reads it as the hive | `/run/credstore/hive-squid.selfcheck` (tmpfs, root, `0400`), imported by `squid-selfcheck.service` as a systemd credential and handed to `curl` on its stdin. Missing, or different from the users file: the self-check logs `credential=absent` or a 407 and fails, and nothing else breaks | | the squid self-check's password | `swarm-controller` mints `swarm/hives/<hive>/squid-selfcheck`; `squid-selfcheck-credential` (root, every 10 minutes) reads it as the hive | `/run/credstore/hive-squid.selfcheck` (tmpfs, root, `0400`), imported by `squid-selfcheck.service` as a systemd credential and handed to `curl` on its stdin. Both self-check users (`selfcheck.squid` and the always-unrestricted `selfcheck-unrestricted.squid`) use it, so whoever holds it has unrestricted egress through this squid Missing, or different from the users file: the self-check logs `credential=absent` or a 407 and fails, and nothing else breaks |
### Store identities, one per principal ### Store identities, one per principal

4
docs/swarm/ui.md
View file

@ -568,7 +568,9 @@ the swarm never set the flag, which counts as on. The **restrict** or
**allow unrestricted** badge flips it with a `PUT` that carries the stored **allow unrestricted** badge flips it with a `PUT` that carries the stored
permission list along with the flag, because a `PUT` without the list resets a permission list along with the flag, because a `PUT` without the list resets a
stored list to the default. An agent with no stored list sends the flag alone. stored list to the default. An agent with no stored list sends the flag alone.
Nothing acts on the flag yet. The hive's egress proxy applies the flag to each agent's proxy credential (see
[the egress proxy](../networking/network.md#egress-proxy-squid)); no agent sends
one yet, so the flag changes no traffic today.
A saved change isn't instant. The hive reads the caps about every 30 seconds, A saved change isn't instant. The hive reads the caps about every 30 seconds,
and a running agent takes a changed cap at its next spawn or rebuild, so the and a running agent takes a changed cap at its next spawn or rebuild, so the

5
nix/host-modules/hive-c0re/environment.nix
View file

@ -272,8 +272,11 @@ in
BAO_CLIENT_CERT = "%d/bao-client.pem"; BAO_CLIENT_CERT = "%d/bao-client.pem";
BAO_CLIENT_KEY = "%d/bao-client-key.pem"; BAO_CLIENT_KEY = "%d/bao-client-key.pem";
# Where `workers::squid_users` renders the hashes of this hive's agents' proxy # Where `workers::squid_users` renders the hashes of this hive's agents' proxy
# credentials for squid; it reads them from the store under the identity above. # credentials for squid, and the list of the users whose agent holds the
# swarm's `unrestricted_internet` flag; it reads the credentials from the
# store under the identity above.
HIVE_SQUID_USERS_FILE = config.services.hyperhive.squid.usersFile; HIVE_SQUID_USERS_FILE = config.services.hyperhive.squid.usersFile;
HIVE_SQUID_UNRESTRICTED_FILE = config.services.hyperhive.squid.unrestrictedFile;
} }
// lib.optionalAttrs (haveBaoClientIdentity && baoDeploy.serverCaFile != null) { // lib.optionalAttrs (haveBaoClientIdentity && baoDeploy.serverCaFile != null) {
# Absent means the system trust store — right for a deployment with a real # Absent means the system trust store — right for a deployment with a real

119
nix/host-modules/hive-squid.nix
View file

@ -1,9 +1,12 @@
# Egress proxy for this hive's agents: squid on the host, bound to the bridge # Egress proxy for this hive's agents: squid on the host, bound to the bridge
# IP, default-deny on the destination hostname. It filters CONNECT targets and # IP, default-deny on the destination hostname. It filters CONNECT targets and
# plain-HTTP hosts and never decrypts, so it holds no signing key. A request # plain-HTTP hosts and never decrypts, so it holds no signing key. A request
# for a listed name must carry a per-agent proxy credential, checked against the # must carry a per-agent proxy credential, checked against the users file
# users file hive-c0re renders. No agent sends one yet; the exporter, journal # hive-c0re renders. A credential's agent either has the swarm's
# forwarding and the self-check below are how a deploy shows it works. # `unrestricted_internet` flag, and may then reach any destination the deny
# rules leave, or reaches only the listed names. No agent sends a credential
# yet; the exporter, journal forwarding and the self-check below are how a
# deploy shows it works.
{ {
pkgs, pkgs,
lib, lib,
@ -24,18 +27,21 @@ let
# The squid user the self-check logs in as: `swarm_secret_client::proxy::SELFCHECK_USER`. # The squid user the self-check logs in as: `swarm_secret_client::proxy::SELFCHECK_USER`.
selfcheckUser = "selfcheck.squid"; selfcheckUser = "selfcheck.squid";
# The same string in `swarm_secret_client::proxy::SELFCHECK_UNRESTRICTED_USER`.
selfcheckUnrestrictedUser = "selfcheck-unrestricted.squid";
usersDir = builtins.dirOf cfg.usersFile; usersDir = builtins.dirOf cfg.usersFile;
# Evaluated top to bottom, first match wins. The three destination denies # Evaluated top to bottom, first match wins. The three destination denies
# sit before the only allow: squid resolves names on the host, so a name # sit before every allow, `allow unrestricted` included: squid resolves
# that answers 127.0.0.1, 169.254.169.254 or a bridge address would # names on the host, so a name that answers 127.0.0.1, 169.254.169.254 or a
# otherwise reach services the bridge firewall keeps agents out of. The # bridge address would otherwise reach services the bridge firewall keeps
# unlisted-name deny comes first because `dstdomain` needs no lookup while # agents out of, and an unrestricted agent is still an agent. The IP-literal
# the `dst` ACLs do: an unlisted name is refused at once instead of waiting # denies come first because `dstdomain` on an IP literal does a reverse
# on the host resolver, and costs a client no DNS query. The credential # lookup. The credential check sits right before the first allow, so only a
# check sits right before the allow, so a refused destination answers 403 # request that would otherwise be let through gets the 407 challenge. An
# to a client with or without a credential, and only a request that would # unrestricted user is then let through to any other destination on a safe
# otherwise be let through gets the 407 challenge. # port, and every other user to the listed names. The `dst` denies cost an
# unlisted name a lookup on the host's resolver before it is refused.
squidConf = '' squidConf = ''
http_port ${netCfg.bridgeIp}:${toString port} http_port ${netCfg.bridgeIp}:${toString port}
pid_filename /run/squid/squid.pid pid_filename /run/squid/squid.pid
@ -59,6 +65,10 @@ let
acl SSL_ports port 443 acl SSL_ports port 443
acl Safe_ports port 80 443 acl Safe_ports port 80 443
acl authed proxy_auth REQUIRED acl authed proxy_auth REQUIRED
# The users whose agent holds the flag. Squid reads the file when it parses
# its configuration, so `squid-unrestricted-reload` reloads squid when
# hive-c0re rewrites it. An empty or missing file lists no one.
acl unrestricted proxy_auth "${cfg.unrestrictedFile}"
# Host-originated requests: the self-check and the exporter. A local # Host-originated requests: the self-check and the exporter. A local
# connect to the bridge IP takes the bridge IP as its source. # connect to the bridge IP takes the bridge IP as its source.
@ -82,11 +92,12 @@ let
http_access deny CONNECT !SSL_ports http_access deny CONNECT !SSL_ports
http_access deny ip_v4_host http_access deny ip_v4_host
http_access deny ip_v6_host http_access deny ip_v6_host
${lib.optionalString (allowed != [ ]) "http_access deny !allowed_dst"}
http_access deny to_localhost http_access deny to_localhost
http_access deny to_linklocal http_access deny to_linklocal
http_access deny to_bridge http_access deny to_bridge
http_access deny !authed http_access deny !authed
http_access allow unrestricted
${lib.optionalString (allowed != [ ]) "http_access deny !allowed_dst"}
${lib.optionalString (allowed != [ ]) "http_access allow allowed_dst"} ${lib.optionalString (allowed != [ ]) "http_access allow allowed_dst"}
http_access deny all http_access deny all
''; '';
@ -128,30 +139,32 @@ let
secret=$(< "$credential_file") secret=$(< "$credential_file")
fi fi
# <host> <auth|anon>: the status squid answers a CONNECT with; 000 when # <host> <user|anon>: the status squid answers a CONNECT with; 000 when
# the proxy is down. # the proxy is down. Both users log in with the one secret.
connect_status() { connect_status() {
{ {
if [ "$2" = auth ] && [ -n "$secret" ]; then if [ "$2" != anon ] && [ -n "$secret" ]; then
printf 'proxy-user = "%s:%s"\n' ${selfcheckUser} "$secret" printf 'proxy-user = "%s:%s"\n' "$2" "$secret"
fi fi
} | curl -sS -m 10 -K - --interface ${netCfg.bridgeIp} -p -x ${proxyUrl} \ } | curl -sS -m 10 -K - --interface ${netCfg.bridgeIp} -p -x ${proxyUrl} \
-o /dev/null -w '%{http_connect}' "https://$1/" 2>/dev/null || true -o /dev/null -w '%{http_connect}' "https://$1/" 2>/dev/null || true
} }
allow_status=$(connect_status ${cfg.selfcheck.canary} auth) allow_status=$(connect_status ${cfg.selfcheck.canary} ${selfcheckUser})
deny_status=$(connect_status ${denyCanary} auth) deny_status=$(connect_status ${denyCanary} ${selfcheckUser})
anon_status=$(connect_status ${cfg.selfcheck.canary} anon) anon_status=$(connect_status ${cfg.selfcheck.canary} anon)
unrestricted_status=$(connect_status ${cfg.selfcheck.unrestrictedCanary} ${selfcheckUnrestrictedUser})
ok=false ok=false
if [ "$allow_status" = 200 ] && [ "$deny_status" = 403 ] && [ "$anon_status" = 407 ]; then if [ "$allow_status" = 200 ] && [ "$deny_status" = 403 ] && [ "$anon_status" = 407 ] \
&& [ "$unrestricted_status" = 200 ]; then
ok=true ok=true
fi fi
credential=absent credential=absent
if [ -n "$secret" ]; then if [ -n "$secret" ]; then
credential=present credential=present
fi fi
echo "squid-selfcheck ok=$ok allow_canary=${cfg.selfcheck.canary} allow_status=$allow_status deny_canary=${denyCanary} deny_status=$deny_status anon_status=$anon_status credential=$credential" echo "squid-selfcheck ok=$ok allow_canary=${cfg.selfcheck.canary} allow_status=$allow_status deny_canary=${denyCanary} deny_status=$deny_status anon_status=$anon_status credential=$credential unrestricted_canary=${cfg.selfcheck.unrestrictedCanary} unrestricted_status=$unrestricted_status"
[ "$ok" = true ] [ "$ok" = true ]
''; '';
}; };
@ -186,6 +199,23 @@ in
''; '';
}; };
unrestrictedFile = lib.mkOption {
type = lib.types.path;
default = "${builtins.dirOf cfg.usersFile}/unrestricted";
readOnly = true;
defaultText = lib.literalMD "`unrestricted`, in the directory of `usersFile`";
description = ''
The user names, one per line, that squid lets reach any destination the
deny rules leave open: the users of the agents that hold the swarm's
`unrestricted_internet` flag (set unless the swarm records it off), and
the self-check's unrestricted user. hive-c0re renders it beside
`usersFile`, and squid is reloaded when it changes. Every other user
reaches only `allowedDomains`. An empty file, which is what exists
until hive-c0re writes one, makes every user a restricted one.
Owned by `hive-core` and readable by the `squid` group.
'';
};
selfcheck.credentialName = lib.mkOption { selfcheck.credentialName = lib.mkOption {
type = lib.types.str; type = lib.types.str;
default = "hive-squid.selfcheck"; default = "hive-squid.selfcheck";
@ -198,6 +228,17 @@ in
''; '';
}; };
selfcheck.unrestrictedCanary = lib.mkOption {
type = lib.types.str;
default = "example.com";
description = ''
Hostname outside `allowedDomains` that the self-check CONNECTs to on
port 443, as its unrestricted user, and expects squid to let through.
Squid resolves it with this host's resolver and connects to it, so a
host with no route to it reports `ok=false`.
'';
};
selfcheck.canary = lib.mkOption { selfcheck.canary = lib.mkOption {
type = lib.types.str; type = lib.types.str;
default = hyperhiveCfg.swarm.forge.domain; default = hyperhiveCfg.swarm.forge.domain;
@ -230,11 +271,43 @@ in
}; };
# The directory is setgid so the file hive-c0re renames into it is the # The directory is setgid so the file hive-c0re renames into it is the
# squid group's. The file exists before squid starts, empty: the helper # squid group's. The users file exists before squid starts, empty: the
# exits on a missing file and squid then rejects every login. # helper exits on a missing file and squid then rejects every login. The
# unrestricted list is created the same way, though squid treats a missing
# one as empty.
systemd.tmpfiles.rules = [ systemd.tmpfiles.rules = [
"d ${usersDir} 2750 hive-core squid - -" "d ${usersDir} 2750 hive-core squid - -"
"f ${cfg.usersFile} 0640 hive-core squid - -" "f ${cfg.usersFile} 0640 hive-core squid - -"
"f ${cfg.unrestrictedFile} 0640 hive-core squid - -"
];
# Squid reads the unrestricted list when it parses its configuration, not
# when it changes. hive-c0re writes the list only when its contents change,
# so each change is one reload. Nothing happens to a squid that is not
# running; it reads the list when it starts.
systemd.paths.squid-unrestricted-reload = {
description = "Watch squid's unrestricted-users list";
wantedBy = [ "paths.target" ];
pathConfig.PathChanged = cfg.unrestrictedFile;
};
systemd.services.squid-unrestricted-reload = {
description = "Reload squid to read its unrestricted-users list";
serviceConfig = {
Type = "oneshot";
ExecStart = "${config.systemd.package}/bin/systemctl try-reload-or-restart squid.service";
};
};
assertions = [
{
assertion = !(builtins.elem cfg.selfcheck.unrestrictedCanary allowed);
message = ''
services.hyperhive.squid.selfcheck.unrestrictedCanary must be a name outside
allowedDomains, or the self-check cannot tell an unrestricted user from a
restricted one.
'';
}
]; ];
systemd.services.squid = { systemd.services.squid = {

68
nix/module-eval/hive-otel.nix
View file

@ -485,38 +485,80 @@ let
&& !(lib.hasInfix "localnet" conf); && !(lib.hasInfix "localnet" conf);
} }
{ {
# First match wins, so a destination deny after the allow is dead, and # First match wins, so a destination deny after an allow is dead, and a
# a credential check after it never challenges anyone. The two IP-literal # credential check after one never challenges anyone. `allow unrestricted`
# denies stay ahead of the unlisted-name deny: `dstdomain` on an IP # is an allow: above the three destination denies an unrestricted user
# reaches loopback, link-local and the bridge through the proxy. The two
# IP-literal denies stay ahead of every allow: `dstdomain` on an IP
# literal does a reverse lookup. # literal does a reverse lookup.
name = "squid refuses unlisted names, then loopback, link-local and bridge destinations, then a missing credential, before any allow"; name = "squid refuses loopback, link-local and bridge destinations and IP literals, then a missing credential, before any allow";
ok = ok =
let let
lines = lib.splitString "\n" hiveOtel.services.squid.configText; lines = lib.splitString "\n" hiveOtel.services.squid.configText;
at = l: lib.lists.findFirstIndex (x: x == l) null lines; at = l: lib.lists.findFirstIndex (x: x == l) null lines;
allow = at "http_access allow allowed_dst"; allow = at "http_access allow allowed_dst";
unrestricted = at "http_access allow unrestricted";
authed = at "http_access deny !authed"; authed = at "http_access deny !authed";
firstAllow = if unrestricted < allow then unrestricted else allow;
in in
allow != null allow != null
&& unrestricted != null
&& authed != null && authed != null
&& at "http_access deny to_localhost" != null && at "http_access deny to_localhost" != null
&& at "http_access deny to_localhost" < allow
&& at "http_access deny to_linklocal" != null && at "http_access deny to_linklocal" != null
&& at "http_access deny to_linklocal" < allow
&& at "http_access deny to_bridge" != null && at "http_access deny to_bridge" != null
&& at "http_access deny to_bridge" < allow && at "http_access deny ip_v4_host" != null
&& at "http_access deny ip_v6_host" != null
&& lib.all (l: at l < firstAllow) [
"http_access deny to_localhost"
"http_access deny to_linklocal"
"http_access deny to_bridge"
"http_access deny ip_v4_host"
"http_access deny ip_v6_host"
]
&& at "http_access deny to_localhost" < authed && at "http_access deny to_localhost" < authed
&& at "http_access deny to_linklocal" < authed && at "http_access deny to_linklocal" < authed
&& at "http_access deny to_bridge" < authed && at "http_access deny to_bridge" < authed
&& authed < allow && authed < firstAllow
&& at "http_access deny ip_v4_host" != null
&& at "http_access deny ip_v4_host" < at "http_access deny !allowed_dst"
&& at "http_access deny ip_v6_host" < at "http_access deny !allowed_dst"
&& at "http_access deny !allowed_dst" != null
&& at "http_access deny !allowed_dst" < at "http_access deny to_localhost"
&& builtins.elem "cache deny all" lines && builtins.elem "cache deny all" lines
&& lib.last (lib.filter (l: l != "") lines) == "http_access deny all"; && lib.last (lib.filter (l: l != "") lines) == "http_access deny all";
} }
{
# An unrestricted user is let through before the unlisted-name deny and
# everyone else is held to the listed names after it. Control: both are
# there, so the ordering below is not a lookup of nothing.
name = "squid lets an unrestricted user through before the allow-list rules that bind everyone else";
ok =
let
conf = hiveOtel.services.squid.configText;
lines = lib.splitString "\n" conf;
at = l: lib.lists.findFirstIndex (x: x == l) null lines;
list = hiveOtel.services.hyperhive.squid.unrestrictedFile;
in
at "http_access allow unrestricted" != null
&& at "http_access deny !allowed_dst" != null
&& at "http_access allow unrestricted" < at "http_access deny !allowed_dst"
&& at "http_access deny !allowed_dst" < at "http_access allow allowed_dst"
&& builtins.elem ''acl unrestricted proxy_auth "${list}"'' lines
&& list == "/var/lib/hive-squid/unrestricted";
}
{
# The list is read when squid parses its configuration, so a change to
# the file reloads squid. Control: the watched path is the list's own.
name = "squid is reloaded when its unrestricted-users list changes";
ok =
let
list = hiveOtel.services.hyperhive.squid.unrestrictedFile;
rules = hiveOtel.systemd.tmpfiles.rules;
path = hiveOtel.systemd.paths.squid-unrestricted-reload;
svc = hiveOtel.systemd.services.squid-unrestricted-reload.serviceConfig;
in
path.pathConfig.PathChanged == list
&& builtins.elem "paths.target" path.wantedBy
&& lib.hasSuffix "systemctl try-reload-or-restart squid.service" svc.ExecStart
&& builtins.elem "f ${list} 0640 hive-core squid - -" rules
&& hiveOtelAllLocal.systemd.services.hive-c0re.environment.HIVE_SQUID_UNRESTRICTED_FILE == list;
}
{ {
# The helper is the NCSA one, on the file hive-c0re renders. The file # The helper is the NCSA one, on the file hive-c0re renders. The file
# exists, empty, before squid starts (the helper exits on a missing one), # exists, empty, before squid starts (the helper exits on a missing one),

153
nix/vm-tests/hive-squid.nix
View file

@ -112,13 +112,18 @@ pkgs.testers.runNixOSTest {
]; ];
# Names the proxy resolves, each a different destination class. # Names the proxy resolves, each a different destination class.
networking.hosts = { networking.hosts = {
"192.168.2.2" = [ "forge.t.local" ]; # `other.t.local` resolves, is reachable and is on no allow-list.
"192.168.2.2" = [
"forge.t.local"
"other.t.local"
];
"127.0.0.1" = [ "loop.t.local" ]; "127.0.0.1" = [ "loop.t.local" ];
${bridgeIp} = [ "bridge.t.local" ]; ${bridgeIp} = [ "bridge.t.local" ];
"169.254.169.254" = [ "meta.t.local" ]; "169.254.169.254" = [ "meta.t.local" ];
}; };
# Renders the users file the way hive-c0re does. # Renders the users file the way hive-c0re does.
environment.systemPackages = [ pkgs.mkpasswd ]; environment.systemPackages = [ pkgs.mkpasswd ];
services.hyperhive.squid.selfcheck.unrestrictedCanary = "other.t.local";
services.hyperhive.squid.allowedDomains = [ services.hyperhive.squid.allowedDomains = [
"forge.t.local" "forge.t.local"
"loop.t.local" "loop.t.local"
@ -157,14 +162,24 @@ pkgs.testers.runNixOSTest {
testScript = '' testScript = ''
import re import re
import secrets import secrets
import time
# Drawn fresh each run. The users file holds only their hashes, as # Drawn fresh each run. The users file holds only their hashes, as
# hive-c0re renders it. # hive-c0re renders it. `atlas` is a restricted agent and `uma` an
PW = {u: secrets.token_urlsafe(24) for u in ("atlas", "selfcheck.squid", "ruth")} # unrestricted one; the two self-check users share one secret.
PW = {
u: secrets.token_urlsafe(24)
for u in ("atlas", "uma", "selfcheck.squid", "ruth")
}
PW["selfcheck-unrestricted.squid"] = PW["selfcheck.squid"]
ATLAS = "atlas:" + PW["atlas"] ATLAS = "atlas:" + PW["atlas"]
UMA = "uma:" + PW["uma"]
SELFCHECK = "selfcheck.squid:" + PW["selfcheck.squid"] SELFCHECK = "selfcheck.squid:" + PW["selfcheck.squid"]
SELFCHECK_U = "selfcheck-unrestricted.squid:" + PW["selfcheck.squid"]
RUTH = "ruth:" + PW["ruth"] RUTH = "ruth:" + PW["ruth"]
USERS = "/var/lib/hive-squid/users" USERS = "/var/lib/hive-squid/users"
LIST = "/var/lib/hive-squid/unrestricted"
UNRESTRICTED = ["uma", "selfcheck-unrestricted.squid"]
start_all() start_all()
proxy.wait_for_unit("squid.service") proxy.wait_for_unit("squid.service")
@ -194,17 +209,42 @@ pkgs.testers.runNixOSTest {
f"&& chmod 640 {USERS}.new && mv {USERS}.new {USERS} && sleep 1" f"&& chmod 640 {USERS}.new && mv {USERS}.new {USERS} && sleep 1"
) )
with subtest("squid starts on the empty users file and refuses every login"): def write_list(*names):
# What hive-c0re does for the list. Squid reads it on a reload, which
# squid-unrestricted-reload does when the file changes.
body = "printf '%s\\n' " + " ".join(names) if names else ":"
proxy.succeed(
f"({body}) > {LIST}.new && chown hive-core:squid {LIST}.new "
f"&& chmod 640 {LIST}.new && mv {LIST}.new {LIST}"
)
def wait_status(host, cred, want):
# The reload that follows a list write is asynchronous.
got = None
for _ in range(30):
got = connect(host, cred)
if got == want:
return
time.sleep(1)
raise AssertionError(f"{host} as {cred.split(':')[0]}: wanted {want}, last got {got}")
with subtest("squid starts on the empty users file and list and refuses every login"):
assert proxy.succeed(f"stat -c '%U:%G %a' {USERS}").strip() == "hive-core:squid 640" assert proxy.succeed(f"stat -c '%U:%G %a' {USERS}").strip() == "hive-core:squid 640"
assert proxy.succeed(f"stat -c '%U:%G %a' {LIST}").strip() == "hive-core:squid 640"
assert proxy.succeed("stat -c %a /var/lib/hive-squid").strip() == "2750" assert proxy.succeed("stat -c %a /var/lib/hive-squid").strip() == "2750"
assert proxy.succeed(f"stat -c %s {USERS}").strip() == "0" assert proxy.succeed(f"stat -c %s {USERS}").strip() == "0"
assert proxy.succeed(f"stat -c %s {LIST}").strip() == "0"
assert connect("forge.t.local", ATLAS) == "407", "no such user" assert connect("forge.t.local", ATLAS) == "407", "no such user"
assert connect("deny-canary.invalid", ATLAS) == "403", "an unlisted name needs no login" assert connect("other.t.local", UMA) == "407", "no such user, unlisted name"
with subtest("the users file holds hashes, never a password"): with subtest("the users file holds hashes, never a password"):
write_users(ATLAS, SELFCHECK) write_users(ATLAS, UMA, SELFCHECK, SELFCHECK_U)
users = proxy.succeed(f"cat {USERS}") users = proxy.succeed(f"cat {USERS}")
assert re.fullmatch(r"(atlas|selfcheck\.squid):\$6\$\S+\n(atlas|selfcheck\.squid):\$6\$\S+\n", users), users rows = [l.split(":", 1) for l in users.splitlines()]
assert sorted(r[0] for r in rows) == sorted(
["atlas", "uma", "selfcheck.squid", "selfcheck-unrestricted.squid"]
), users
assert all(re.fullmatch(r"\$6\$\S+", r[1]) for r in rows), users
assert not any(p in users for p in PW.values()), users assert not any(p in users for p in PW.values()), users
with subtest("an allowed name is tunnelled with a credential and challenged without one"): with subtest("an allowed name is tunnelled with a credential and challenged without one"):
@ -215,22 +255,31 @@ pkgs.testers.runNixOSTest {
# DES would accept the first eight characters; the hash must not. # DES would accept the first eight characters; the hash must not.
assert connect("forge.t.local", "atlas:" + PW["atlas"][:8]) == "407", "a prefix of the password" assert connect("forge.t.local", "atlas:" + PW["atlas"][:8]) == "407", "a prefix of the password"
with subtest("an unlisted name is refused with or without a credential"): with subtest("with an empty list every user is restricted"):
assert connect("deny-canary.invalid", ATLAS) == "403", "deny canary" assert connect("other.t.local", UMA) == "403", "uma, not listed yet"
assert connect("deny-canary.invalid") == "403", "deny canary, anonymous" assert connect("other.t.local", ATLAS) == "403"
assert connect("forge.t.local", UMA) == "200", "control: uma still reaches a listed name"
with subtest("a name resolving to a host-local destination is refused"): with subtest("a listed user reaches an unlisted name and the others are refused"):
assert connect("loop.t.local", ATLAS) == "403", "loopback" write_list(*UNRESTRICTED)
assert connect("bridge.t.local", ATLAS) == "403", "bridge address" wait_status("other.t.local", UMA, "200")
assert connect("meta.t.local", ATLAS) == "403", "link-local" assert connect("other.t.local", ATLAS) == "403", "restricted"
assert connect("forge.t.local", ATLAS) == "200", "restricted, listed name"
assert connect("forge.t.local", UMA) == "200", "unrestricted, listed name"
assert connect("other.t.local") == "407", "no credential"
assert connect("other.t.local", "uma:" + secrets.token_urlsafe(24)) == "407", "wrong password"
assert connect("deny-canary.invalid", ATLAS) == "403", "restricted, a name that does not resolve"
with subtest("IP literals are refused even when the address is reachable"): with subtest("neither kind of user reaches a host-local destination or another port"):
assert connect("127.0.0.1", ATLAS) == "403" for cred in (ATLAS, UMA):
assert connect("192.168.2.2", ATLAS) == "403" who = cred.split(":")[0]
assert connect("loop.t.local", cred) == "403", f"{who}: loopback"
with subtest("only a port-443 CONNECT is allowed"): assert connect("bridge.t.local", cred) == "403", f"{who}: bridge address"
assert connect("forge.t.local:22", ATLAS) == "403" assert connect("meta.t.local", cred) == "403", f"{who}: link-local"
assert connect("forge.t.local:80", ATLAS) == "403" assert connect("127.0.0.1", cred) == "403", f"{who}: loopback literal"
assert connect("192.168.2.2", cred) == "403", f"{who}: IP literal"
assert connect("other.t.local:22", cred) == "403", f"{who}: port 22"
assert connect("other.t.local:80", cred) == "403", f"{who}: CONNECT to port 80"
with subtest("an agent cannot read cachemgr"): with subtest("an agent cannot read cachemgr"):
agent.succeed( agent.succeed(
@ -247,32 +296,50 @@ pkgs.testers.runNixOSTest {
"journalctl -u squid-selfcheck.service -o cat | " "journalctl -u squid-selfcheck.service -o cat | "
"grep -F 'squid-selfcheck ok=true allow_canary=forge.t.local " "grep -F 'squid-selfcheck ok=true allow_canary=forge.t.local "
"allow_status=200 deny_canary=deny-canary.invalid deny_status=403 " "allow_status=200 deny_canary=deny-canary.invalid deny_status=403 "
"anon_status=407 credential=present'" "anon_status=407 credential=present "
"unrestricted_canary=other.t.local unrestricted_status=200'"
) )
with subtest("the self-check fails, and says why, when the unrestricted user is not listed"):
write_list()
wait_status("other.t.local", UMA, "403")
proxy.fail("systemctl start squid-selfcheck.service")
proxy.succeed(
"journalctl -u squid-selfcheck.service -o cat | "
"grep -F 'squid-selfcheck ok=false allow_canary=forge.t.local "
"allow_status=200 deny_canary=deny-canary.invalid deny_status=403 "
"anon_status=407 credential=present "
"unrestricted_canary=other.t.local unrestricted_status=403'"
)
write_list(*UNRESTRICTED)
wait_status("other.t.local", UMA, "200")
with subtest("the self-check fails, and says why, without a credential or with the wrong one"): with subtest("the self-check fails, and says why, without a credential or with the wrong one"):
proxy.succeed("mv /run/credstore/hive-squid.selfcheck /run/credstore/kept") proxy.succeed("mv /run/credstore/hive-squid.selfcheck /run/credstore/kept")
proxy.fail("systemctl start squid-selfcheck.service") proxy.fail("systemctl start squid-selfcheck.service")
proxy.succeed( proxy.succeed(
"journalctl -u squid-selfcheck.service -o cat | " "journalctl -u squid-selfcheck.service -o cat | "
"grep -F 'squid-selfcheck ok=false allow_canary=forge.t.local " "grep -F 'squid-selfcheck ok=false allow_canary=forge.t.local "
"allow_status=407 deny_canary=deny-canary.invalid deny_status=403 " "allow_status=407 deny_canary=deny-canary.invalid deny_status=407 "
"anon_status=407 credential=absent'" "anon_status=407 credential=absent "
"unrestricted_canary=other.t.local unrestricted_status=407'"
) )
proxy.succeed(f"printf '%s\\n' {secrets.token_urlsafe(24)} > /run/credstore/hive-squid.selfcheck") proxy.succeed(f"printf '%s\\n' {secrets.token_urlsafe(24)} > /run/credstore/hive-squid.selfcheck")
proxy.fail("systemctl start squid-selfcheck.service") proxy.fail("systemctl start squid-selfcheck.service")
proxy.succeed( proxy.succeed(
"journalctl -u squid-selfcheck.service -o cat | " "journalctl -u squid-selfcheck.service -o cat | "
"grep -F 'allow_status=407 deny_canary=deny-canary.invalid deny_status=403 " "grep -F 'allow_status=407 deny_canary=deny-canary.invalid deny_status=407 "
"anon_status=407 credential=present'" "anon_status=407 credential=present "
"unrestricted_canary=other.t.local unrestricted_status=407'"
) )
proxy.succeed("mv -f /run/credstore/kept /run/credstore/hive-squid.selfcheck") proxy.succeed("mv -f /run/credstore/kept /run/credstore/hive-squid.selfcheck")
with subtest("squid's access log reaches the journal under squid.service, with the user"): with subtest("squid's access log reaches the journal under squid.service, with the user"):
journal = proxy.succeed("journalctl -u squid.service -o cat") journal = proxy.succeed("journalctl -u squid.service -o cat")
assert re.search(r"192\.168\.1\.3 TCP_TUNNEL/200 .* CONNECT forge\.t\.local:443 atlas ", journal), journal assert re.search(r"192\.168\.1\.3 TCP_TUNNEL/200 .* CONNECT forge\.t\.local:443 atlas ", journal), journal
assert re.search(r"192\.168\.1\.3 TCP_TUNNEL/200 .* CONNECT other\.t\.local:443 uma ", journal), journal
assert re.search(r"192\.168\.1\.3 TCP_DENIED/407 .* CONNECT forge\.t\.local:443 - ", journal), journal assert re.search(r"192\.168\.1\.3 TCP_DENIED/407 .* CONNECT forge\.t\.local:443 - ", journal), journal
assert re.search(r"192\.168\.1\.3 TCP_DENIED/403 .* CONNECT deny-canary\.invalid:443 ", journal), journal assert re.search(r"192\.168\.1\.3 TCP_DENIED/403 .* CONNECT other\.t\.local:443 atlas ", journal), journal
assert re.search(r"192\.168\.1\.3 TCP_DENIED/403 .* CONNECT loop\.t\.local:443 ", journal), journal assert re.search(r"192\.168\.1\.3 TCP_DENIED/403 .* CONNECT loop\.t\.local:443 ", journal), journal
# Exporter scrapes are cachemgr requests and stay out of the log. # Exporter scrapes are cachemgr requests and stay out of the log.
assert "squid-internal-mgr" not in journal, journal assert "squid-internal-mgr" not in journal, journal
@ -287,15 +354,36 @@ pkgs.testers.runNixOSTest {
assert m and float(m.group(1)) >= 1, metrics assert m and float(m.group(1)) >= 1, metrics
proxy.fail("curl -sf -m 5 http://${bridgeIp}:9301/metrics") proxy.fail("curl -sf -m 5 http://${bridgeIp}:9301/metrics")
with subtest("squid survives a reload"): with subtest("squid survives a reload and keeps the list"):
proxy.succeed("systemctl reload squid.service") proxy.succeed("systemctl reload squid.service")
proxy.wait_for_unit("squid.service") proxy.wait_for_unit("squid.service")
assert connect("forge.t.local", ATLAS) == "200" assert connect("forge.t.local", ATLAS) == "200"
assert connect("other.t.local", UMA) == "200"
assert connect("other.t.local", ATLAS) == "403"
with subtest("a user added while squid runs can log in, with no reload"): with subtest("a user added while squid runs can log in, with no reload, and is restricted"):
write_users(ATLAS, SELFCHECK, RUTH) write_users(ATLAS, UMA, SELFCHECK, SELFCHECK_U, RUTH)
assert connect("forge.t.local", RUTH) == "200" assert connect("forge.t.local", RUTH) == "200"
assert connect("forge.t.local", ATLAS) == "200" assert connect("forge.t.local", ATLAS) == "200"
assert connect("other.t.local", RUTH) == "403"
with subtest("a user the list gains is unrestricted after the reload and loses it again"):
write_list(*UNRESTRICTED, "ruth")
wait_status("other.t.local", RUTH, "200")
assert connect("loop.t.local", RUTH) == "403", "still no loopback"
write_list(*UNRESTRICTED)
wait_status("other.t.local", RUTH, "403")
assert connect("other.t.local", UMA) == "200", "control: uma is still listed"
with subtest("squid starts on a missing list, treats every user as restricted, and recovers when it appears"):
proxy.succeed(f"rm {LIST}")
proxy.succeed("systemctl restart squid.service")
proxy.wait_for_unit("squid.service")
proxy.wait_for_open_port(3128, "${bridgeIp}")
assert connect("forge.t.local", UMA) == "200", "logins work"
assert connect("other.t.local", UMA) == "403", "no list, no unrestricted user"
write_list(*UNRESTRICTED)
wait_status("other.t.local", UMA, "200")
with subtest("squid starts on a missing users file, refuses logins, and recovers when it appears"): with subtest("squid starts on a missing users file, refuses logins, and recovers when it appears"):
proxy.succeed(f"rm {USERS}") proxy.succeed(f"rm {USERS}")
@ -303,8 +391,9 @@ pkgs.testers.runNixOSTest {
proxy.wait_for_unit("squid.service") proxy.wait_for_unit("squid.service")
proxy.wait_for_open_port(3128, "${bridgeIp}") proxy.wait_for_open_port(3128, "${bridgeIp}")
assert connect("forge.t.local", ATLAS) == "407", "no file, no login" assert connect("forge.t.local", ATLAS) == "407", "no file, no login"
assert connect("deny-canary.invalid", ATLAS) == "403", "an unlisted name still answers" assert connect("other.t.local", UMA) == "407", "no file, no login, even when listed"
write_users(ATLAS, SELFCHECK) write_users(ATLAS, UMA, SELFCHECK, SELFCHECK_U)
assert connect("forge.t.local", ATLAS) == "200", "the file appeared" assert connect("forge.t.local", ATLAS) == "200", "the file appeared"
assert connect("other.t.local", UMA) == "200", "listed, and now logged in"
''; '';
} }