Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
86e9f49fe6 | ||
|
|
e8868d52a8 | ||
|
|
3d9561a9a6 | ||
|
|
501445ebc3 | ||
|
|
0b99f17bc3 |
10 changed files with 827 additions and 162 deletions
|
|
@ -230,18 +230,46 @@ host. It never decrypts TLS, so it holds no signing key. It caches nothing.
|
|||
store and the next pass mints a new one. HTTP Basic crosses the bridge in
|
||||
clear between an agent and the proxy. See
|
||||
[the credentials reference](../swarm/credentials-reference.md).
|
||||
A name that isn't on the allow-list gets the 403 whether or not the request
|
||||
has a credential; a listed name without one gets a 407.
|
||||
A request without a valid credential gets a 407, once the destination
|
||||
checks below have passed it.
|
||||
|
||||
- **Two lists.** The credential decides the policy. `hive-c0re` writes a
|
||||
second file beside the users file, `services.hyperhive.squid.unrestrictedFile`
|
||||
(`/var/lib/hive-squid/unrestricted`, `hive-core:squid`, `0640`): the names of
|
||||
the users whose agent holds the `unrestricted_internet` flag in the
|
||||
permissions the hive last read from the swarm, one per line, rewritten when
|
||||
it changes. The default is permissive: an agent with no record, or a record
|
||||
that never set the flag, holds it, the same default as the
|
||||
`hyperhive.agent.unrestricted_internet` metric. While the hive's read of the
|
||||
permissions is stale or failing, a new agent that already holds a
|
||||
credential counts as unrestricted. Squid reads the list when it
|
||||
parses its configuration, so `squid-unrestricted-reload.path` reloads squid
|
||||
when the file changes. A missing or empty file lists no one, and squid
|
||||
starts either way. A flag change reaches squid after the hive's next read of
|
||||
the permissions and its next 30-second pass.
|
||||
An older `hive-c0re` writes no list, so every credential on that hive is
|
||||
restricted. Agents must not start sending credentials until every hive runs
|
||||
the `hive-c0re` that writes it, or a legacy agent loses general internet.
|
||||
|
||||
- **Allow-list.** `services.hyperhive.squid.allowedDomains` defaults to the
|
||||
swarm's own service names (`services.hyperhive.swarm.serviceDomains`). Each
|
||||
entry matches that exact name. Everything else gets a 403.
|
||||
- **Unlisted names.** Refused first, without a DNS lookup.
|
||||
entry matches that exact name. A user on the unrestricted list reaches any
|
||||
other name too, on ports 80 and 443. Everyone else gets a 403 for it.
|
||||
- **Unlisted names.** For a user who isn't on the unrestricted list, refused
|
||||
after the destination checks, which look the name up on the host.
|
||||
- **Denied before any allow.** Destinations that resolve to loopback,
|
||||
link-local or the bridge subnet, and IP-literal hosts. Squid resolves names
|
||||
on the host, so without these a listed name pointing at `127.0.0.1` would
|
||||
walk around the bridge→loopback DROP rule. Only `CONNECT` to port 443 and
|
||||
plain HTTP to ports 80 and 443 pass.
|
||||
link-local, the bridge subnet, a private range (`10.0.0.0/8`,
|
||||
`172.16.0.0/12`, `192.168.0.0/16`, `fc00::/7`), shared address space
|
||||
(`100.64.0.0/10`), `0.0.0.0/8`, `::`, multicast or reserved space, and IP-literal
|
||||
hosts, for every user including the unrestricted ones and for listed names.
|
||||
Squid resolves names on the host and connects from it, so without these a
|
||||
name pointing at `127.0.0.1`, at the host's own LAN address or at a private
|
||||
neighbour would walk around the bridge→loopback DROP rule and the swarm's
|
||||
network. Squid checks the address it connects to, so a name that changes its
|
||||
answer between a check and the connect gains nothing. A listed name must
|
||||
therefore resolve to a public address on the host's resolver: a split-horizon
|
||||
name that answers a LAN address gets a 403, and so does the self-check
|
||||
canary. Only `CONNECT` to port 443 and plain HTTP to ports 80 and 443 pass.
|
||||
- **Metrics.** `squid-exporter` serves `prometheus-squid-exporter` on
|
||||
`127.0.0.1:9301`. The hive collector scrapes it as job `squid` through
|
||||
`services.hyperhive.otel.scrapeTargets`. `squid_up` and
|
||||
|
|
@ -254,9 +282,13 @@ host. It never decrypts TLS, so it holds no signing key. It caches nothing.
|
|||
every minute from the bridge IP, logging in as `selfcheck.squid`: `CONNECT`
|
||||
to `services.hyperhive.squid.selfcheck.canary` (default: the forge's name)
|
||||
must answer 200, to `deny-canary.invalid` 403, and to the canary without a
|
||||
credential 407. It writes one `squid-selfcheck ok=… allow_status=…
|
||||
deny_status=… anon_status=… credential=present|absent` line per run and
|
||||
fails the unit when any is wrong. The password is the hive's
|
||||
credential 407. It then logs in as `selfcheck-unrestricted.squid`, which has
|
||||
the same secret and is always on the unrestricted list, and a `CONNECT` to
|
||||
`services.hyperhive.squid.selfcheck.unrestrictedCanary` (default:
|
||||
`example.com`, a name outside the allow-list) must answer 200. It writes one
|
||||
`squid-selfcheck ok=… allow_status=… deny_status=… anon_status=…
|
||||
credential=present|absent unrestricted_canary=… unrestricted_status=…` line
|
||||
per run and fails the unit when any is wrong. The password is the hive's
|
||||
`swarm/hives/<hive>/squid-selfcheck` secret, which
|
||||
`squid-selfcheck-credential` lands at `/run/credstore/hive-squid.selfcheck`.
|
||||
Before `swarm-controller` has minted it, or while `hive-c0re` hasn't
|
||||
|
|
|
|||
|
|
@ -320,7 +320,10 @@ an absent flag included, and a value in the request wins. A `PUT` that leaves
|
|||
the field out and a `DELETE` (`swarmctl agent permissions clear`) also keep the
|
||||
stored value; a `DELETE` on an agent that has one rewrites the record to just
|
||||
that value. Read it with
|
||||
`swarm_queue_client::agent_permissions::unrestricted`. Nothing acts on it yet.
|
||||
`swarm_queue_client::agent_permissions::unrestricted`. The hive reads it for
|
||||
the agent's squid policy: [the egress proxy](../networking/network.md#egress-proxy-squid)
|
||||
lets an unrestricted agent's credential reach any destination, and holds the
|
||||
others to the allow-list. No agent sends a credential yet.
|
||||
|
||||
**How a change applies** —
|
||||
- The agent's harness reads its own key every 10 seconds and builds every
|
||||
|
|
|
|||
|
|
@ -148,16 +148,16 @@ pieces fit, and the reasoning behind them, is in [`credentials.md`](credentials.
|
|||
|
||||
### Hive-level — one of each per hive
|
||||
|
||||
| secret | generated by | lives at |
|
||||
| ---------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
|
||||
| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` |
|
||||
| the cache's public key and pull token, for hive-priv | `swarm-controller` copies them to `swarm/hives/<hive>/attic/{public-key,pull-token}` (see the cache-facts row above) | `hive-priv-cache-facts` (root) reads those two under the hive's own store identity every ten minutes and writes `/run/hyperhive-priv/attic/public-key` and `netrc` (tmpfs, root, each `0600`, directory `0700`); the cache's address comes from `services.hyperhive.swarm.attic.domain` and `cache`, not from the store. A store that holds no facts, or that doesn't answer, leaves the files as they stand, and a hive that never got them has `hive-priv` answer `not available`. |
|
||||
| the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/run/hive-forge-oidc/<id>.secret` on the host (tmpfs), bound into the forge container, handed to `forgejo-sso-source` through `LoadCredential` |
|
||||
| the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/<id>.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` |
|
||||
| this hive's forge read token | `swarm-controller`, into the secret store at `swarm/hives/<hive>/forge/token`, for the `hive-<hive>` account; a pass at start and every five minutes re-mints a missing or stale token | `hive-c0re` reads it under the hive's own store identity, and copies it (account on the first line, token on the second) to `/run/hyperhive/forge-hive-token` (tmpfs, `0600` `hive-core`) for nix's `git`, which `hive-priv` also runs. Each `internal/meta` read and each boot rewrites the copy when it differs; where they differ after a rotation, a build that starts before the next read fails to authenticate, and a retry reads the new token |
|
||||
| squid's users file | `hive-c0re` (`workers::squid_users`), every 30 seconds, from the proxy credentials of its agents and the self-check | `services.hyperhive.squid.usersFile` (`/var/lib/hive-squid/users`, `hive-core:squid`, `0640`): `<user>:<hash>` lines, SHA-512 crypt hashes only, renamed into place when it changed. `systemd-tmpfiles` creates it empty so squid's helper never sees a missing file. A hive whose agents have no credential yet (an older controller) has an empty file: every login fails and the self-check reports `ok=false`; no agent sends a credential, so no traffic changes |
|
||||
| the squid self-check's password | `swarm-controller` mints `swarm/hives/<hive>/squid-selfcheck`; `squid-selfcheck-credential` (root, every 10 minutes) reads it as the hive | `/run/credstore/hive-squid.selfcheck` (tmpfs, root, `0400`), imported by `squid-selfcheck.service` as a systemd credential and handed to `curl` on its stdin. Missing, or different from the users file: the self-check logs `credential=absent` or a 407 and fails, and nothing else breaks |
|
||||
| secret | generated by | lives at |
|
||||
| ---------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
|
||||
| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` |
|
||||
| the cache's public key and pull token, for hive-priv | `swarm-controller` copies them to `swarm/hives/<hive>/attic/{public-key,pull-token}` (see the cache-facts row above) | `hive-priv-cache-facts` (root) reads those two under the hive's own store identity every ten minutes and writes `/run/hyperhive-priv/attic/public-key` and `netrc` (tmpfs, root, each `0600`, directory `0700`); the cache's address comes from `services.hyperhive.swarm.attic.domain` and `cache`, not from the store. A store that holds no facts, or that doesn't answer, leaves the files as they stand, and a hive that never got them has `hive-priv` answer `not available`. |
|
||||
| the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/run/hive-forge-oidc/<id>.secret` on the host (tmpfs), bound into the forge container, handed to `forgejo-sso-source` through `LoadCredential` |
|
||||
| the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/<id>.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` |
|
||||
| this hive's forge read token | `swarm-controller`, into the secret store at `swarm/hives/<hive>/forge/token`, for the `hive-<hive>` account; a pass at start and every five minutes re-mints a missing or stale token | `hive-c0re` reads it under the hive's own store identity, and copies it (account on the first line, token on the second) to `/run/hyperhive/forge-hive-token` (tmpfs, `0600` `hive-core`) for nix's `git`, which `hive-priv` also runs. Each `internal/meta` read and each boot rewrites the copy when it differs; where they differ after a rotation, a build that starts before the next read fails to authenticate, and a retry reads the new token |
|
||||
| squid's users file | `hive-c0re` (`workers::squid_users`), every 30 seconds, from the proxy credentials of its agents and the self-check | `services.hyperhive.squid.usersFile` (`/var/lib/hive-squid/users`, `hive-core:squid`, `0640`): `<user>:<hash>` lines, SHA-512 crypt hashes only, renamed into place when it changed. `unrestricted` beside it lists the users whose agent holds the swarm's `unrestricted_internet` flag, names only; it lists every agent the hive never read as off `systemd-tmpfiles` creates it empty so squid's helper never sees a missing file. A hive whose agents have no credential yet (an older controller) has an empty file: every login fails and the self-check reports `ok=false`; no agent sends a credential, so no traffic changes |
|
||||
| the squid self-check's password | `swarm-controller` mints `swarm/hives/<hive>/squid-selfcheck`; `squid-selfcheck-credential` (root, every 10 minutes) reads it as the hive | `/run/credstore/hive-squid.selfcheck` (tmpfs, root, `0400`), imported by `squid-selfcheck.service` as a systemd credential and handed to `curl` on its stdin. Both self-check users (`selfcheck.squid` and the always-unrestricted `selfcheck-unrestricted.squid`) use it, so whoever holds it has unrestricted egress through this squid Missing, or different from the users file: the self-check logs `credential=absent` or a 407 and fails, and nothing else breaks |
|
||||
|
||||
### Store identities, one per principal
|
||||
|
||||
|
|
|
|||
|
|
@ -568,7 +568,9 @@ the swarm never set the flag, which counts as on. The **restrict** or
|
|||
**allow unrestricted** badge flips it with a `PUT` that carries the stored
|
||||
permission list along with the flag, because a `PUT` without the list resets a
|
||||
stored list to the default. An agent with no stored list sends the flag alone.
|
||||
Nothing acts on the flag yet.
|
||||
The hive's egress proxy applies the flag to each agent's proxy credential (see
|
||||
[the egress proxy](../networking/network.md#egress-proxy-squid)); no agent sends
|
||||
one yet, so the flag changes no traffic today.
|
||||
|
||||
A saved change isn't instant. The hive reads the caps about every 30 seconds,
|
||||
and a running agent takes a changed cap at its next spawn or rebuild, so the
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
//! Squid's users file: the hashes of the proxy credentials this hive's agents
|
||||
//! authenticate to its egress proxy with.
|
||||
//! Squid's users file, and its list of unrestricted users: the hashes of the
|
||||
//! proxy credentials this hive's agents authenticate to its egress proxy with,
|
||||
//! and which of them may reach any destination.
|
||||
//!
|
||||
//! swarm-controller mints one credential per agent and one for this hive's own
|
||||
//! squid self-check ([`swarm_secret_client::proxy`]). Every [`INTERVAL`] this
|
||||
|
|
@ -9,12 +10,19 @@
|
|||
//! moves. The agents are the container list, not the running containers: a
|
||||
//! stopped agent stays in the file.
|
||||
//!
|
||||
//! Only salted SHA-512 crypt hashes reach the disk. A hash is kept as it is
|
||||
//! while it still verifies its credential, so the file changes only when a
|
||||
//! user is added, removed or given a new credential.
|
||||
//! The file named by [`ENV_UNRESTRICTED_FILE`] lists, one name per line, the
|
||||
//! users whose agent holds `unrestricted_internet` in the permissions this
|
||||
//! hive last read from the swarm (an agent with no record, or no flag in it,
|
||||
//! holds it), plus the self-check's unrestricted user. Squid reads the list as
|
||||
//! an ACL when it parses its configuration, so `nix/host-modules/hive-squid.nix`
|
||||
//! reloads squid when the file changes.
|
||||
//!
|
||||
//! A failed read leaves the file as it is: dropping a user on a store outage
|
||||
//! would read exactly like the swarm revoking it.
|
||||
//! Only salted SHA-512 crypt hashes reach the disk. A hash is kept as it is
|
||||
//! while it still verifies its credential, so each file changes only when a
|
||||
//! user is added, removed, given a new credential or has its flag flipped.
|
||||
//!
|
||||
//! A failed read leaves both files as they are: dropping a user on a store
|
||||
//! outage would read exactly like the swarm revoking it.
|
||||
|
||||
use std::future::Future;
|
||||
use std::path::{Path, PathBuf};
|
||||
|
|
@ -24,6 +32,7 @@ use anyhow::{Context, Result, bail};
|
|||
use sha_crypt::{PasswordHasher as _, PasswordVerifier as _, ShaCrypt};
|
||||
use swarm_secret_client::{SecretStore, client, proxy};
|
||||
|
||||
use crate::permissions;
|
||||
use crate::stats::sweep_health::{self, SweepHealth};
|
||||
|
||||
/// How often the file is re-rendered.
|
||||
|
|
@ -36,11 +45,15 @@ const FAILURES_BEFORE_WARNING: u32 = 3;
|
|||
/// from `services.hyperhive.squid.usersFile`; unset means this host runs no squid.
|
||||
pub const ENV_USERS_FILE: &str = "HIVE_SQUID_USERS_FILE";
|
||||
|
||||
/// The environment variable naming the unrestricted-users list. Set by
|
||||
/// `nix/host-modules/hive-c0re` from `services.hyperhive.squid.unrestrictedFile`.
|
||||
pub const ENV_UNRESTRICTED_FILE: &str = "HIVE_SQUID_UNRESTRICTED_FILE";
|
||||
|
||||
/// Raw salt bytes: twelve of them encode to the sixteen characters `crypt()`
|
||||
/// accepts.
|
||||
const SALT_BYTES: usize = 12;
|
||||
|
||||
/// The file's mode. The group is squid's, from the setgid directory
|
||||
/// Both files' mode. The group is squid's, from the setgid directory
|
||||
/// `nix/host-modules/hive-squid.nix` creates.
|
||||
const FILE_MODE: u32 = 0o640;
|
||||
|
||||
|
|
@ -49,14 +62,32 @@ const FILE_MODE: u32 = 0o640;
|
|||
struct User {
|
||||
name: String,
|
||||
secret: String,
|
||||
/// Whether squid lets this user reach any destination the deny rules leave.
|
||||
unrestricted: bool,
|
||||
}
|
||||
|
||||
/// Start the pass loop, when this host has a squid users file, a store and a
|
||||
/// hive name.
|
||||
pub fn spawn(mut shutdown: tokio::sync::watch::Receiver<bool>) {
|
||||
let Some(path) = std::env::var_os(ENV_USERS_FILE)
|
||||
/// The two files a pass writes.
|
||||
struct Files<'a> {
|
||||
users: &'a Path,
|
||||
unrestricted: &'a Path,
|
||||
}
|
||||
|
||||
/// Whether `name` is one of the self-check's squid users rather than an agent's.
|
||||
fn is_selfcheck(name: &str) -> bool {
|
||||
name == proxy::SELFCHECK_USER || name == proxy::SELFCHECK_UNRESTRICTED_USER
|
||||
}
|
||||
|
||||
fn env_path(name: &str) -> Option<PathBuf> {
|
||||
std::env::var_os(name)
|
||||
.filter(|v| !v.is_empty())
|
||||
.map(PathBuf::from)
|
||||
}
|
||||
|
||||
/// Start the pass loop, when this host has squid's two files, a store and a
|
||||
/// hive name.
|
||||
pub fn spawn(mut shutdown: tokio::sync::watch::Receiver<bool>) {
|
||||
let (Some(users), Some(unrestricted)) =
|
||||
(env_path(ENV_USERS_FILE), env_path(ENV_UNRESTRICTED_FILE))
|
||||
else {
|
||||
return;
|
||||
};
|
||||
|
|
@ -73,7 +104,15 @@ pub fn spawn(mut shutdown: tokio::sync::watch::Receiver<bool>) {
|
|||
tokio::spawn(async move {
|
||||
let mut health = SweepHealth::new("squid_users_render", "warn", FAILURES_BEFORE_WARNING);
|
||||
loop {
|
||||
match pass(&hive, &path).await {
|
||||
match pass(
|
||||
&hive,
|
||||
&Files {
|
||||
users: &users,
|
||||
unrestricted: &unrestricted,
|
||||
},
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(()) => health.record_ok(),
|
||||
Err(e) => {
|
||||
tracing::warn!(error = ?e, "squid users: the pass failed");
|
||||
|
|
@ -99,18 +138,21 @@ pub fn spawn(mut shutdown: tokio::sync::watch::Receiver<bool>) {
|
|||
});
|
||||
}
|
||||
|
||||
/// Read this hive's credentials from the store and render them into `file`.
|
||||
async fn pass(hive: &str, file: &Path) -> Result<()> {
|
||||
/// Read this hive's credentials from the store and the swarm's permissions
|
||||
/// from the last read of them, and render both files.
|
||||
async fn pass(hive: &str, files: &Files<'_>) -> Result<()> {
|
||||
let agents: Vec<String> = crate::lifecycle::agents_for_meta_listing()
|
||||
.await
|
||||
.context("enumerating this hive's agents")?
|
||||
.into_iter()
|
||||
.map(|spec| spec.name)
|
||||
.collect();
|
||||
let perms = permissions::read_at(&permissions::path())
|
||||
.context("reading the stored swarm permissions")?;
|
||||
let store = SecretStore::from_env(hive)
|
||||
.await
|
||||
.context("connecting to the store")?;
|
||||
sync_with(hive, &agents, file, |path| {
|
||||
sync_with(hive, &agents, &perms, files, |path| {
|
||||
let store = &store;
|
||||
async move {
|
||||
let held: Option<proxy::Credential> = store
|
||||
|
|
@ -123,49 +165,104 @@ async fn pass(hive: &str, file: &Path) -> Result<()> {
|
|||
.await
|
||||
}
|
||||
|
||||
/// [`pass`] with the store read and the file given. Nothing is written unless
|
||||
/// every read succeeds and the rendered file differs from the current one.
|
||||
async fn sync_with<F, Fut>(hive: &str, agents: &[String], file: &Path, read: F) -> Result<()>
|
||||
/// [`pass`] with the store read, the permissions and the files given. Nothing
|
||||
/// is written unless every read succeeds, and a file is only written when its
|
||||
/// rendering differs from what it holds.
|
||||
///
|
||||
/// The unrestricted list goes first. A user that turns restricted is then
|
||||
/// restricted before the users file stops listing it, and one that turns
|
||||
/// unrestricted is not listed before it can log in.
|
||||
async fn sync_with<F, Fut>(
|
||||
hive: &str,
|
||||
agents: &[String],
|
||||
perms: &permissions::Stored,
|
||||
files: &Files<'_>,
|
||||
read: F,
|
||||
) -> Result<()>
|
||||
where
|
||||
F: Fn(String) -> Fut,
|
||||
Fut: Future<Output = Result<Option<String>>>,
|
||||
{
|
||||
let mut wanted = Vec::new();
|
||||
for agent in agents {
|
||||
wanted.push((agent.clone(), proxy::agent_credential_path(agent)?));
|
||||
let unrestricted = permissions::unrestricted_internet(perms.get(agent));
|
||||
wanted.push((
|
||||
agent.clone(),
|
||||
proxy::agent_credential_path(agent)?,
|
||||
unrestricted,
|
||||
));
|
||||
}
|
||||
let selfcheck = proxy::selfcheck_path(hive)?;
|
||||
wanted.push((proxy::SELFCHECK_USER.to_owned(), selfcheck.clone(), false));
|
||||
wanted.push((
|
||||
proxy::SELFCHECK_USER.to_owned(),
|
||||
proxy::selfcheck_path(hive)?,
|
||||
proxy::SELFCHECK_UNRESTRICTED_USER.to_owned(),
|
||||
selfcheck,
|
||||
true,
|
||||
));
|
||||
let mut users = Vec::new();
|
||||
for (name, path) in wanted {
|
||||
for (name, path, unrestricted) in wanted {
|
||||
if let Some(secret) = read(path).await? {
|
||||
users.push(User { name, secret });
|
||||
users.push(User {
|
||||
name,
|
||||
secret,
|
||||
unrestricted,
|
||||
});
|
||||
} else {
|
||||
tracing::debug!(name, "squid users: no proxy credential minted yet");
|
||||
}
|
||||
}
|
||||
let current = match std::fs::read_to_string(file) {
|
||||
Ok(text) => text,
|
||||
Err(e) if e.kind() == std::io::ErrorKind::NotFound => String::new(),
|
||||
Err(e) => return Err(e).with_context(|| format!("reading {}", file.display())),
|
||||
};
|
||||
let list = render_unrestricted(&users);
|
||||
if write_if_changed(files.unrestricted, &list)? {
|
||||
tracing::info!(
|
||||
unrestricted = list.lines().count(),
|
||||
file = %files.unrestricted.display(),
|
||||
"squid users: wrote the unrestricted list"
|
||||
);
|
||||
}
|
||||
let current = read_or_empty(files.users)?;
|
||||
let rendered = render(¤t, &users)?;
|
||||
if rendered == current {
|
||||
return Ok(());
|
||||
if write_if_changed(files.users, &rendered)? {
|
||||
tracing::info!(
|
||||
users = users.len(),
|
||||
agents = users.iter().filter(|u| !is_selfcheck(&u.name)).count(),
|
||||
file = %files.users.display(),
|
||||
"squid users: wrote the users file"
|
||||
);
|
||||
}
|
||||
crate::secret_file::write_secret_mode(file, rendered.as_bytes(), FILE_MODE)
|
||||
.with_context(|| format!("writing {}", file.display()))?;
|
||||
tracing::info!(
|
||||
users = users.len(),
|
||||
agents = users.iter().filter(|u| u.name != proxy::SELFCHECK_USER).count(),
|
||||
file = %file.display(),
|
||||
"squid users: wrote the users file"
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// The file's text, or the empty string when it does not exist.
|
||||
fn read_or_empty(file: &Path) -> Result<String> {
|
||||
match std::fs::read_to_string(file) {
|
||||
Ok(text) => Ok(text),
|
||||
Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(String::new()),
|
||||
Err(e) => Err(e).with_context(|| format!("reading {}", file.display())),
|
||||
}
|
||||
}
|
||||
|
||||
/// Write `text` to `file` unless it already holds exactly that. Returns
|
||||
/// whether it wrote.
|
||||
fn write_if_changed(file: &Path, text: &str) -> Result<bool> {
|
||||
if read_or_empty(file)? == text {
|
||||
return Ok(false);
|
||||
}
|
||||
crate::secret_file::write_secret_mode(file, text.as_bytes(), FILE_MODE)
|
||||
.with_context(|| format!("writing {}", file.display()))?;
|
||||
Ok(true)
|
||||
}
|
||||
|
||||
/// The names of the unrestricted `users`, sorted, one per line.
|
||||
fn render_unrestricted(users: &[User]) -> String {
|
||||
let mut names: Vec<&str> = users
|
||||
.iter()
|
||||
.filter(|u| u.unrestricted)
|
||||
.map(|u| u.name.as_str())
|
||||
.collect();
|
||||
names.sort_unstable();
|
||||
names.iter().flat_map(|n| [n, "\n"]).collect()
|
||||
}
|
||||
|
||||
/// The users file for `users`, sorted by name, one `<user>:<hash>` line each.
|
||||
///
|
||||
/// A user whose line in `current` still verifies against its credential keeps
|
||||
|
|
@ -215,6 +312,7 @@ mod tests {
|
|||
use std::os::unix::fs::PermissionsExt as _;
|
||||
|
||||
use super::*;
|
||||
use swarm_wire::agent_permissions::AgentPermissions;
|
||||
|
||||
const HIVE: &str = "h1";
|
||||
|
||||
|
|
@ -230,14 +328,54 @@ mod tests {
|
|||
names.iter().map(|n| (*n).to_owned()).collect()
|
||||
}
|
||||
|
||||
async fn sync(held: &HashMap<String, String>, agents: &[&str], file: &Path) -> Result<()> {
|
||||
sync_with(HIVE, &names(agents), file, |path| {
|
||||
/// The unrestricted list that sits beside the users file `users`.
|
||||
fn list_of(users: &Path) -> PathBuf {
|
||||
users.with_file_name("unrestricted")
|
||||
}
|
||||
|
||||
/// The permissions record of an agent whose swarm flag is `flag`.
|
||||
fn flagged(agent: &str, flag: Option<bool>) -> (String, AgentPermissions) {
|
||||
let json = match flag {
|
||||
Some(f) => format!(r#"{{"unrestricted_internet":{f}}}"#),
|
||||
None => "{}".to_owned(),
|
||||
};
|
||||
(
|
||||
agent.to_owned(),
|
||||
serde_json::from_str(&json).expect("decodes"),
|
||||
)
|
||||
}
|
||||
|
||||
async fn sync_flags(
|
||||
held: &HashMap<String, String>,
|
||||
agents: &[&str],
|
||||
perms: &permissions::Stored,
|
||||
file: &Path,
|
||||
) -> Result<()> {
|
||||
let list = list_of(file);
|
||||
let files = Files {
|
||||
users: file,
|
||||
unrestricted: &list,
|
||||
};
|
||||
sync_with(HIVE, &names(agents), perms, &files, |path| {
|
||||
let found = held.get(&path).cloned();
|
||||
async move { Ok(found) }
|
||||
})
|
||||
.await
|
||||
}
|
||||
|
||||
/// [`sync_flags`] with no permissions read yet: every agent unrestricted.
|
||||
async fn sync(held: &HashMap<String, String>, agents: &[&str], file: &Path) -> Result<()> {
|
||||
sync_flags(held, agents, &permissions::Stored::new(), file).await
|
||||
}
|
||||
|
||||
fn list(users: &Path) -> Vec<String> {
|
||||
std::fs::read_to_string(list_of(users))
|
||||
.expect("the list is there")
|
||||
.lines()
|
||||
.map(str::to_owned)
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn lines(file: &Path) -> Vec<(String, String)> {
|
||||
std::fs::read_to_string(file)
|
||||
.expect("the file is there")
|
||||
|
|
@ -267,7 +405,15 @@ mod tests {
|
|||
.await
|
||||
.expect("renders");
|
||||
let users: Vec<String> = lines(&file).into_iter().map(|(n, _)| n).collect();
|
||||
assert_eq!(users, ["argus", "atlas", "selfcheck.squid"]);
|
||||
assert_eq!(
|
||||
users,
|
||||
[
|
||||
"argus",
|
||||
"atlas",
|
||||
"selfcheck-unrestricted.squid",
|
||||
"selfcheck.squid"
|
||||
]
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
|
|
@ -298,17 +444,17 @@ mod tests {
|
|||
sync(&two_agents_and_a_stranger(), &["atlas", "argus"], &file)
|
||||
.await
|
||||
.expect("renders");
|
||||
// Control: the search finds a name that is in the file.
|
||||
// Control: the search finds a name that is in each file.
|
||||
let mut found = 0;
|
||||
for entry in std::fs::read_dir(dir.path()).expect("lists") {
|
||||
let text = std::fs::read_to_string(entry.expect("entry").path()).expect("text");
|
||||
assert!(text.contains("atlas:"), "control: {text}");
|
||||
assert!(text.contains("atlas"), "control: {text}");
|
||||
for secret in ["atlas-secret", "argus-secret", "check-secret"] {
|
||||
assert!(!text.contains(secret), "{secret} reached the disk");
|
||||
}
|
||||
found += 1;
|
||||
}
|
||||
assert_eq!(found, 1, "only the users file is left behind");
|
||||
assert_eq!(found, 2, "only the users file and the list are left behind");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
|
|
@ -390,7 +536,10 @@ mod tests {
|
|||
std::fs::write(&file, text).expect("write");
|
||||
sync(&held, &["atlas"], &file).await.expect("renders");
|
||||
let users: Vec<String> = lines(&file).into_iter().map(|(n, _)| n).collect();
|
||||
assert_eq!(users, ["atlas", "selfcheck.squid"]);
|
||||
assert_eq!(
|
||||
users,
|
||||
["atlas", "selfcheck-unrestricted.squid", "selfcheck.squid"]
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
|
|
@ -411,10 +560,17 @@ mod tests {
|
|||
.await
|
||||
.expect("renders");
|
||||
let before = std::fs::read_to_string(&file).expect("read");
|
||||
let list_before = std::fs::read_to_string(list_of(&file)).expect("read");
|
||||
let list = list_of(&file);
|
||||
let files = Files {
|
||||
users: &file,
|
||||
unrestricted: &list,
|
||||
};
|
||||
let failed = sync_with(
|
||||
HIVE,
|
||||
&names(&["atlas", "argus"]),
|
||||
&file,
|
||||
&permissions::Stored::from([flagged("atlas", Some(false))]),
|
||||
&files,
|
||||
|path| async move {
|
||||
if path.contains("argus") {
|
||||
bail!("the store is down")
|
||||
|
|
@ -425,6 +581,11 @@ mod tests {
|
|||
.await;
|
||||
assert!(failed.is_err());
|
||||
assert_eq!(std::fs::read_to_string(&file).expect("read"), before);
|
||||
assert_eq!(
|
||||
std::fs::read_to_string(&list).expect("read"),
|
||||
list_before,
|
||||
"the flag flip waits for every read"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
|
@ -433,8 +594,124 @@ mod tests {
|
|||
let user = User {
|
||||
name: bad.to_owned(),
|
||||
secret: "s".to_owned(),
|
||||
unrestricted: false,
|
||||
};
|
||||
assert!(render("", &[user]).is_err(), "{bad:?}");
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn an_agent_is_listed_unless_its_flag_is_off() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let file = dir.path().join("users");
|
||||
let held = store(&[
|
||||
("swarm/agents/old/proxy-credential", "s1"),
|
||||
("swarm/agents/unset/proxy-credential", "s2"),
|
||||
("swarm/agents/on/proxy-credential", "s3"),
|
||||
("swarm/agents/new/proxy-credential", "s4"),
|
||||
("swarm/hives/h1/squid-selfcheck", "check-secret"),
|
||||
]);
|
||||
let perms = permissions::Stored::from([
|
||||
flagged("unset", None),
|
||||
flagged("on", Some(true)),
|
||||
flagged("new", Some(false)),
|
||||
]);
|
||||
sync_flags(&held, &["old", "unset", "on", "new"], &perms, &file)
|
||||
.await
|
||||
.expect("renders");
|
||||
// `old` has no record at all, `unset` a record without the flag.
|
||||
assert_eq!(
|
||||
list(&file),
|
||||
["old", "on", "selfcheck-unrestricted.squid", "unset"]
|
||||
);
|
||||
// Control: the restricted agent still has a login.
|
||||
let users: Vec<String> = lines(&file).into_iter().map(|(n, _)| n).collect();
|
||||
assert!(users.contains(&"new".to_owned()), "{users:?}");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn the_selfchecks_log_in_with_one_secret_and_only_one_is_unrestricted() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let file = dir.path().join("users");
|
||||
sync(&two_agents_and_a_stranger(), &["atlas"], &file)
|
||||
.await
|
||||
.expect("renders");
|
||||
let by_name: HashMap<_, _> = lines(&file).into_iter().collect();
|
||||
assert!(verifies("check-secret", &by_name["selfcheck.squid"]));
|
||||
assert!(verifies(
|
||||
"check-secret",
|
||||
&by_name["selfcheck-unrestricted.squid"]
|
||||
));
|
||||
let listed = list(&file);
|
||||
assert!(listed.contains(&"selfcheck-unrestricted.squid".to_owned()));
|
||||
assert!(
|
||||
!listed.contains(&"selfcheck.squid".to_owned()),
|
||||
"{listed:?}"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn an_agent_with_no_credential_is_not_listed() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let file = dir.path().join("users");
|
||||
sync(&two_agents_and_a_stranger(), &["atlas", "fresh"], &file)
|
||||
.await
|
||||
.expect("renders");
|
||||
let listed = list(&file);
|
||||
assert!(listed.contains(&"atlas".to_owned()), "control: {listed:?}");
|
||||
assert!(!listed.contains(&"fresh".to_owned()), "{listed:?}");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_flipped_flag_rewrites_the_list_and_leaves_the_users_file() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let file = dir.path().join("users");
|
||||
let held = two_agents_and_a_stranger();
|
||||
let ino =
|
||||
|p: &Path| std::os::unix::fs::MetadataExt::ino(&std::fs::metadata(p).expect("stat"));
|
||||
sync(&held, &["atlas", "argus"], &file)
|
||||
.await
|
||||
.expect("renders");
|
||||
assert!(list(&file).contains(&"argus".to_owned()));
|
||||
let (users_ino, list_ino) = (ino(&file), ino(&list_of(&file)));
|
||||
// An unchanged pass touches neither.
|
||||
sync(&held, &["atlas", "argus"], &file)
|
||||
.await
|
||||
.expect("renders");
|
||||
assert_eq!((ino(&file), ino(&list_of(&file))), (users_ino, list_ino));
|
||||
// Turning argus off changes only the list.
|
||||
let perms = permissions::Stored::from([flagged("argus", Some(false))]);
|
||||
sync_flags(&held, &["atlas", "argus"], &perms, &file)
|
||||
.await
|
||||
.expect("renders");
|
||||
assert!(!list(&file).contains(&"argus".to_owned()));
|
||||
assert!(list(&file).contains(&"atlas".to_owned()));
|
||||
assert_eq!(ino(&file), users_ino, "the users file was rewritten");
|
||||
assert_ne!(ino(&list_of(&file)), list_ino, "the list was not");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn the_list_is_group_readable() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let file = dir.path().join("users");
|
||||
sync(&two_agents_and_a_stranger(), &["atlas"], &file)
|
||||
.await
|
||||
.expect("renders");
|
||||
let meta = std::fs::metadata(list_of(&file)).expect("stat");
|
||||
assert_eq!(meta.permissions().mode() & 0o777, 0o640);
|
||||
}
|
||||
|
||||
/// A pass that cannot write the users file has still written the list: a
|
||||
/// user turned restricted loses its access before the users file stops
|
||||
/// listing it.
|
||||
#[tokio::test]
|
||||
async fn the_list_is_written_before_the_users_file() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let file = dir.path().join("users");
|
||||
std::fs::create_dir(&file).expect("a directory where the users file belongs");
|
||||
sync(&two_agents_and_a_stranger(), &["atlas"], &file)
|
||||
.await
|
||||
.expect_err("the users file cannot be read");
|
||||
assert_eq!(list(&file), ["atlas", "selfcheck-unrestricted.squid"]);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -272,8 +272,11 @@ in
|
|||
BAO_CLIENT_CERT = "%d/bao-client.pem";
|
||||
BAO_CLIENT_KEY = "%d/bao-client-key.pem";
|
||||
# Where `workers::squid_users` renders the hashes of this hive's agents' proxy
|
||||
# credentials for squid; it reads them from the store under the identity above.
|
||||
# credentials for squid, and the list of the users whose agent holds the
|
||||
# swarm's `unrestricted_internet` flag; it reads the credentials from the
|
||||
# store under the identity above.
|
||||
HIVE_SQUID_USERS_FILE = config.services.hyperhive.squid.usersFile;
|
||||
HIVE_SQUID_UNRESTRICTED_FILE = config.services.hyperhive.squid.unrestrictedFile;
|
||||
}
|
||||
// lib.optionalAttrs (haveBaoClientIdentity && baoDeploy.serverCaFile != null) {
|
||||
# Absent means the system trust store — right for a deployment with a real
|
||||
|
|
|
|||
|
|
@ -1,9 +1,12 @@
|
|||
# Egress proxy for this hive's agents: squid on the host, bound to the bridge
|
||||
# IP, default-deny on the destination hostname. It filters CONNECT targets and
|
||||
# plain-HTTP hosts and never decrypts, so it holds no signing key. A request
|
||||
# for a listed name must carry a per-agent proxy credential, checked against the
|
||||
# users file hive-c0re renders. No agent sends one yet; the exporter, journal
|
||||
# forwarding and the self-check below are how a deploy shows it works.
|
||||
# must carry a per-agent proxy credential, checked against the users file
|
||||
# hive-c0re renders. A credential's agent either has the swarm's
|
||||
# `unrestricted_internet` flag, and may then reach any destination the deny
|
||||
# rules leave, or reaches only the listed names. No agent sends a credential
|
||||
# yet; the exporter, journal forwarding and the self-check below are how a
|
||||
# deploy shows it works.
|
||||
{
|
||||
pkgs,
|
||||
lib,
|
||||
|
|
@ -24,18 +27,45 @@ let
|
|||
|
||||
# The squid user the self-check logs in as: `swarm_secret_client::proxy::SELFCHECK_USER`.
|
||||
selfcheckUser = "selfcheck.squid";
|
||||
# The same string in `swarm_secret_client::proxy::SELFCHECK_UNRESTRICTED_USER`.
|
||||
selfcheckUnrestrictedUser = "selfcheck-unrestricted.squid";
|
||||
usersDir = builtins.dirOf cfg.usersFile;
|
||||
|
||||
# Evaluated top to bottom, first match wins. The three destination denies
|
||||
# sit before the only allow: squid resolves names on the host, so a name
|
||||
# that answers 127.0.0.1, 169.254.169.254 or a bridge address would
|
||||
# otherwise reach services the bridge firewall keeps agents out of. The
|
||||
# unlisted-name deny comes first because `dstdomain` needs no lookup while
|
||||
# the `dst` ACLs do: an unlisted name is refused at once instead of waiting
|
||||
# on the host resolver, and costs a client no DNS query. The credential
|
||||
# check sits right before the allow, so a refused destination answers 403
|
||||
# to a client with or without a credential, and only a request that would
|
||||
# otherwise be let through gets the 407 challenge.
|
||||
# Destinations that are not the public internet: loopback, link-local, the
|
||||
# bridge, and the private, shared (CGNAT), unspecified, multicast and
|
||||
# reserved ranges. Squid checks the address it connects to, so a name an
|
||||
# agent controls cannot point past them.
|
||||
# `::` is listed because a connect to it lands on loopback; squid's
|
||||
# built-in `to_localhost` has `0.0.0.0/32` and `::1` but not `::`. NAT64
|
||||
# (`64:ff9b::/96`) and 6to4 (`2002::/16`) embed IPv4 addresses and are left
|
||||
# out: they matter only on a host with such a route.
|
||||
privateRanges = [
|
||||
"10.0.0.0/8"
|
||||
"172.16.0.0/12"
|
||||
"192.168.0.0/16"
|
||||
"100.64.0.0/10"
|
||||
"0.0.0.0/8"
|
||||
"224.0.0.0/4"
|
||||
"240.0.0.0/4"
|
||||
"fc00::/7"
|
||||
"ff00::/8"
|
||||
"::/128"
|
||||
];
|
||||
|
||||
# Evaluated top to bottom, first match wins. The destination denies sit
|
||||
# before every allow, `allow unrestricted` included: squid resolves names on
|
||||
# the host and connects from it, so a name that answers 127.0.0.1,
|
||||
# 169.254.169.254, the host's own LAN address or a private neighbour would
|
||||
# otherwise reach services the bridge firewall keeps agents out of, and an
|
||||
# unrestricted agent is still an agent. The same denies bind a listed name,
|
||||
# so a listed name must resolve to a public address on the host's resolver.
|
||||
# The IP-literal
|
||||
# denies come first because `dstdomain` on an IP literal does a reverse
|
||||
# lookup. The credential check sits right before the first allow, so only a
|
||||
# request that would otherwise be let through gets the 407 challenge. An
|
||||
# unrestricted user is then let through to any other destination on a safe
|
||||
# port, and every other user to the listed names. The `dst` denies cost an
|
||||
# unlisted name a lookup on the host's resolver before it is refused.
|
||||
squidConf = ''
|
||||
http_port ${netCfg.bridgeIp}:${toString port}
|
||||
pid_filename /run/squid/squid.pid
|
||||
|
|
@ -59,12 +89,17 @@ let
|
|||
acl SSL_ports port 443
|
||||
acl Safe_ports port 80 443
|
||||
acl authed proxy_auth REQUIRED
|
||||
# The users whose agent holds the flag. Squid reads the file when it parses
|
||||
# its configuration, so `squid-unrestricted-reload` reloads squid when
|
||||
# hive-c0re rewrites it. An empty or missing file lists no one.
|
||||
acl unrestricted proxy_auth "${cfg.unrestrictedFile}"
|
||||
|
||||
# Host-originated requests: the self-check and the exporter. A local
|
||||
# connect to the bridge IP takes the bridge IP as its source.
|
||||
acl selfcheck src ${netCfg.bridgeIp}/32
|
||||
|
||||
acl to_bridge dst ${netCfg.networkCidr}
|
||||
acl to_private dst ${lib.concatStringsSep " " privateRanges}
|
||||
# `dstdomain` on an IP-literal host does a reverse lookup, so a PTR record
|
||||
# an agent controls could match an allowed name.
|
||||
acl ip_v4_host url_regex ^([a-z]+://)?[0-9.]+(:[0-9]+)?(/|$)
|
||||
|
|
@ -82,11 +117,13 @@ let
|
|||
http_access deny CONNECT !SSL_ports
|
||||
http_access deny ip_v4_host
|
||||
http_access deny ip_v6_host
|
||||
${lib.optionalString (allowed != [ ]) "http_access deny !allowed_dst"}
|
||||
http_access deny to_localhost
|
||||
http_access deny to_linklocal
|
||||
http_access deny to_bridge
|
||||
http_access deny to_private
|
||||
http_access deny !authed
|
||||
http_access allow unrestricted
|
||||
${lib.optionalString (allowed != [ ]) "http_access deny !allowed_dst"}
|
||||
${lib.optionalString (allowed != [ ]) "http_access allow allowed_dst"}
|
||||
http_access deny all
|
||||
'';
|
||||
|
|
@ -128,30 +165,32 @@ let
|
|||
secret=$(< "$credential_file")
|
||||
fi
|
||||
|
||||
# <host> <auth|anon>: the status squid answers a CONNECT with; 000 when
|
||||
# the proxy is down.
|
||||
# <host> <user|anon>: the status squid answers a CONNECT with; 000 when
|
||||
# the proxy is down. Both users log in with the one secret.
|
||||
connect_status() {
|
||||
{
|
||||
if [ "$2" = auth ] && [ -n "$secret" ]; then
|
||||
printf 'proxy-user = "%s:%s"\n' ${selfcheckUser} "$secret"
|
||||
if [ "$2" != anon ] && [ -n "$secret" ]; then
|
||||
printf 'proxy-user = "%s:%s"\n' "$2" "$secret"
|
||||
fi
|
||||
} | curl -sS -m 10 -K - --interface ${netCfg.bridgeIp} -p -x ${proxyUrl} \
|
||||
-o /dev/null -w '%{http_connect}' "https://$1/" 2>/dev/null || true
|
||||
}
|
||||
|
||||
allow_status=$(connect_status ${cfg.selfcheck.canary} auth)
|
||||
deny_status=$(connect_status ${denyCanary} auth)
|
||||
allow_status=$(connect_status ${cfg.selfcheck.canary} ${selfcheckUser})
|
||||
deny_status=$(connect_status ${denyCanary} ${selfcheckUser})
|
||||
anon_status=$(connect_status ${cfg.selfcheck.canary} anon)
|
||||
unrestricted_status=$(connect_status ${cfg.selfcheck.unrestrictedCanary} ${selfcheckUnrestrictedUser})
|
||||
|
||||
ok=false
|
||||
if [ "$allow_status" = 200 ] && [ "$deny_status" = 403 ] && [ "$anon_status" = 407 ]; then
|
||||
if [ "$allow_status" = 200 ] && [ "$deny_status" = 403 ] && [ "$anon_status" = 407 ] \
|
||||
&& [ "$unrestricted_status" = 200 ]; then
|
||||
ok=true
|
||||
fi
|
||||
credential=absent
|
||||
if [ -n "$secret" ]; then
|
||||
credential=present
|
||||
fi
|
||||
echo "squid-selfcheck ok=$ok allow_canary=${cfg.selfcheck.canary} allow_status=$allow_status deny_canary=${denyCanary} deny_status=$deny_status anon_status=$anon_status credential=$credential"
|
||||
echo "squid-selfcheck ok=$ok allow_canary=${cfg.selfcheck.canary} allow_status=$allow_status deny_canary=${denyCanary} deny_status=$deny_status anon_status=$anon_status credential=$credential unrestricted_canary=${cfg.selfcheck.unrestrictedCanary} unrestricted_status=$unrestricted_status"
|
||||
[ "$ok" = true ]
|
||||
'';
|
||||
};
|
||||
|
|
@ -168,8 +207,12 @@ in
|
|||
];
|
||||
description = ''
|
||||
Hostnames squid lets a client reach, by CONNECT or plain HTTP. Each
|
||||
entry matches that exact name. Everything else gets a 403. The
|
||||
self-check canary is allowed on top of this list.
|
||||
entry matches that exact name. Everything else gets a 403, except
|
||||
for an unrestricted user. The self-check canary is allowed on top of
|
||||
this list. A name must resolve, on this host's resolver, to a public
|
||||
address: one in a loopback, link-local, private, shared (100.64.0.0/10),
|
||||
multicast or reserved range (or is `::`), or in the bridge subnet, gets a 403 for
|
||||
every user.
|
||||
'';
|
||||
};
|
||||
|
||||
|
|
@ -186,6 +229,23 @@ in
|
|||
'';
|
||||
};
|
||||
|
||||
unrestrictedFile = lib.mkOption {
|
||||
type = lib.types.path;
|
||||
default = "${builtins.dirOf cfg.usersFile}/unrestricted";
|
||||
readOnly = true;
|
||||
defaultText = lib.literalMD "`unrestricted`, in the directory of `usersFile`";
|
||||
description = ''
|
||||
The user names, one per line, that squid lets reach any destination the
|
||||
deny rules leave open: the users of the agents that hold the swarm's
|
||||
`unrestricted_internet` flag (set unless the swarm records it off), and
|
||||
the self-check's unrestricted user. hive-c0re renders it beside
|
||||
`usersFile`, and squid is reloaded when it changes. Every other user
|
||||
reaches only `allowedDomains`. An empty file, which is what exists
|
||||
until hive-c0re writes one, makes every user a restricted one.
|
||||
Owned by `hive-core` and readable by the `squid` group.
|
||||
'';
|
||||
};
|
||||
|
||||
selfcheck.credentialName = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
default = "hive-squid.selfcheck";
|
||||
|
|
@ -198,6 +258,17 @@ in
|
|||
'';
|
||||
};
|
||||
|
||||
selfcheck.unrestrictedCanary = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
default = "example.com";
|
||||
description = ''
|
||||
Hostname outside `allowedDomains` that the self-check CONNECTs to on
|
||||
port 443, as its unrestricted user, and expects squid to let through.
|
||||
Squid resolves it with this host's resolver and connects to it, so a
|
||||
host with no route to it reports `ok=false`.
|
||||
'';
|
||||
};
|
||||
|
||||
selfcheck.canary = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
default = hyperhiveCfg.swarm.forge.domain;
|
||||
|
|
@ -205,9 +276,10 @@ in
|
|||
description = ''
|
||||
Hostname the self-check CONNECTs to on port 443 and expects squid to
|
||||
let through. Squid resolves it with this host's resolver, not the
|
||||
hive's: a host whose resolver answers a loopback or bridge address
|
||||
for it (a single-host swarm with the gateway's `localHostsEntry`) gets
|
||||
a 403, and the self-check reports `ok=false`.
|
||||
hive's: a host whose resolver answers a loopback, bridge or other
|
||||
private address for it (a single-host swarm with the gateway's
|
||||
`localHostsEntry`, or a split-horizon LAN) gets a 403, and the
|
||||
self-check reports `ok=false`.
|
||||
'';
|
||||
};
|
||||
};
|
||||
|
|
@ -230,11 +302,43 @@ in
|
|||
};
|
||||
|
||||
# The directory is setgid so the file hive-c0re renames into it is the
|
||||
# squid group's. The file exists before squid starts, empty: the helper
|
||||
# exits on a missing file and squid then rejects every login.
|
||||
# squid group's. The users file exists before squid starts, empty: the
|
||||
# helper exits on a missing file and squid then rejects every login. The
|
||||
# unrestricted list is created the same way, though squid treats a missing
|
||||
# one as empty.
|
||||
systemd.tmpfiles.rules = [
|
||||
"d ${usersDir} 2750 hive-core squid - -"
|
||||
"f ${cfg.usersFile} 0640 hive-core squid - -"
|
||||
"f ${cfg.unrestrictedFile} 0640 hive-core squid - -"
|
||||
];
|
||||
|
||||
# Squid reads the unrestricted list when it parses its configuration, not
|
||||
# when it changes. hive-c0re writes the list only when its contents change,
|
||||
# so each change is one reload. Nothing happens to a squid that is not
|
||||
# running; it reads the list when it starts.
|
||||
systemd.paths.squid-unrestricted-reload = {
|
||||
description = "Watch squid's unrestricted-users list";
|
||||
wantedBy = [ "paths.target" ];
|
||||
pathConfig.PathChanged = cfg.unrestrictedFile;
|
||||
};
|
||||
|
||||
systemd.services.squid-unrestricted-reload = {
|
||||
description = "Reload squid to read its unrestricted-users list";
|
||||
serviceConfig = {
|
||||
Type = "oneshot";
|
||||
ExecStart = "${config.systemd.package}/bin/systemctl try-reload-or-restart squid.service";
|
||||
};
|
||||
};
|
||||
|
||||
assertions = [
|
||||
{
|
||||
assertion = !(builtins.elem cfg.selfcheck.unrestrictedCanary allowed);
|
||||
message = ''
|
||||
services.hyperhive.squid.selfcheck.unrestrictedCanary must be a name outside
|
||||
allowedDomains, or the self-check cannot tell an unrestricted user from a
|
||||
restricted one.
|
||||
'';
|
||||
}
|
||||
];
|
||||
|
||||
systemd.services.squid = {
|
||||
|
|
|
|||
|
|
@ -485,38 +485,110 @@ let
|
|||
&& !(lib.hasInfix "localnet" conf);
|
||||
}
|
||||
{
|
||||
# First match wins, so a destination deny after the allow is dead, and
|
||||
# a credential check after it never challenges anyone. The two IP-literal
|
||||
# denies stay ahead of the unlisted-name deny: `dstdomain` on an IP
|
||||
# First match wins, so a destination deny after an allow is dead, and a
|
||||
# credential check after one never challenges anyone. `allow unrestricted`
|
||||
# is an allow: above the three destination denies an unrestricted user
|
||||
# reaches loopback, link-local and the bridge through the proxy. The two
|
||||
# IP-literal denies stay ahead of every allow: `dstdomain` on an IP
|
||||
# literal does a reverse lookup.
|
||||
name = "squid refuses unlisted names, then loopback, link-local and bridge destinations, then a missing credential, before any allow";
|
||||
name = "squid refuses loopback, link-local, bridge and private destinations and IP literals, then a missing credential, before any allow";
|
||||
ok =
|
||||
let
|
||||
lines = lib.splitString "\n" hiveOtel.services.squid.configText;
|
||||
at = l: lib.lists.findFirstIndex (x: x == l) null lines;
|
||||
allow = at "http_access allow allowed_dst";
|
||||
unrestricted = at "http_access allow unrestricted";
|
||||
authed = at "http_access deny !authed";
|
||||
firstAllow = if unrestricted < allow then unrestricted else allow;
|
||||
in
|
||||
allow != null
|
||||
&& unrestricted != null
|
||||
&& authed != null
|
||||
&& at "http_access deny to_localhost" != null
|
||||
&& at "http_access deny to_localhost" < allow
|
||||
&& at "http_access deny to_linklocal" != null
|
||||
&& at "http_access deny to_linklocal" < allow
|
||||
&& at "http_access deny to_bridge" != null
|
||||
&& at "http_access deny to_bridge" < allow
|
||||
&& at "http_access deny to_private" != null
|
||||
&& at "http_access deny ip_v4_host" != null
|
||||
&& at "http_access deny ip_v6_host" != null
|
||||
&& lib.all (l: at l < firstAllow) [
|
||||
"http_access deny to_localhost"
|
||||
"http_access deny to_linklocal"
|
||||
"http_access deny to_bridge"
|
||||
"http_access deny to_private"
|
||||
"http_access deny ip_v4_host"
|
||||
"http_access deny ip_v6_host"
|
||||
]
|
||||
&& at "http_access deny to_localhost" < authed
|
||||
&& at "http_access deny to_linklocal" < authed
|
||||
&& at "http_access deny to_bridge" < authed
|
||||
&& authed < allow
|
||||
&& at "http_access deny ip_v4_host" != null
|
||||
&& at "http_access deny ip_v4_host" < at "http_access deny !allowed_dst"
|
||||
&& at "http_access deny ip_v6_host" < at "http_access deny !allowed_dst"
|
||||
&& at "http_access deny !allowed_dst" != null
|
||||
&& at "http_access deny !allowed_dst" < at "http_access deny to_localhost"
|
||||
&& at "http_access deny to_private" < authed
|
||||
&& authed < firstAllow
|
||||
&& builtins.elem "cache deny all" lines
|
||||
&& lib.last (lib.filter (l: l != "") lines) == "http_access deny all";
|
||||
}
|
||||
{
|
||||
# The ranges are what keeps an unrestricted user on the public internet.
|
||||
# Control: the bridge deny and the ACL line are found, so a range missing
|
||||
# below is not a lookup of nothing.
|
||||
name = "squid's private-destination ACL names every private, shared, multicast and reserved range";
|
||||
ok =
|
||||
let
|
||||
lines = lib.splitString "\n" hiveOtel.services.squid.configText;
|
||||
acl = lib.findFirst (l: lib.hasPrefix "acl to_private dst " l) null lines;
|
||||
ranges = lib.splitString " " (lib.removePrefix "acl to_private dst " acl);
|
||||
in
|
||||
acl != null
|
||||
&& builtins.elem "acl to_bridge dst ${hiveOtel.services.hyperhive.network.networkCidr}" lines
|
||||
&&
|
||||
lib.sort lib.lessThan ranges == lib.sort lib.lessThan [
|
||||
"10.0.0.0/8"
|
||||
"172.16.0.0/12"
|
||||
"192.168.0.0/16"
|
||||
"100.64.0.0/10"
|
||||
"0.0.0.0/8"
|
||||
"224.0.0.0/4"
|
||||
"240.0.0.0/4"
|
||||
"fc00::/7"
|
||||
"ff00::/8"
|
||||
"::/128"
|
||||
];
|
||||
}
|
||||
{
|
||||
# An unrestricted user is let through before the unlisted-name deny and
|
||||
# everyone else is held to the listed names after it. Control: both are
|
||||
# there, so the ordering below is not a lookup of nothing.
|
||||
name = "squid lets an unrestricted user through before the allow-list rules that bind everyone else";
|
||||
ok =
|
||||
let
|
||||
conf = hiveOtel.services.squid.configText;
|
||||
lines = lib.splitString "\n" conf;
|
||||
at = l: lib.lists.findFirstIndex (x: x == l) null lines;
|
||||
list = hiveOtel.services.hyperhive.squid.unrestrictedFile;
|
||||
in
|
||||
at "http_access allow unrestricted" != null
|
||||
&& at "http_access deny !allowed_dst" != null
|
||||
&& at "http_access allow unrestricted" < at "http_access deny !allowed_dst"
|
||||
&& at "http_access deny !allowed_dst" < at "http_access allow allowed_dst"
|
||||
&& builtins.elem ''acl unrestricted proxy_auth "${list}"'' lines
|
||||
&& list == "/var/lib/hive-squid/unrestricted";
|
||||
}
|
||||
{
|
||||
# The list is read when squid parses its configuration, so a change to
|
||||
# the file reloads squid. Control: the watched path is the list's own.
|
||||
name = "squid is reloaded when its unrestricted-users list changes";
|
||||
ok =
|
||||
let
|
||||
list = hiveOtel.services.hyperhive.squid.unrestrictedFile;
|
||||
rules = hiveOtel.systemd.tmpfiles.rules;
|
||||
path = hiveOtel.systemd.paths.squid-unrestricted-reload;
|
||||
svc = hiveOtel.systemd.services.squid-unrestricted-reload.serviceConfig;
|
||||
in
|
||||
path.pathConfig.PathChanged == list
|
||||
&& builtins.elem "paths.target" path.wantedBy
|
||||
&& lib.hasSuffix "systemctl try-reload-or-restart squid.service" svc.ExecStart
|
||||
&& builtins.elem "f ${list} 0640 hive-core squid - -" rules
|
||||
&& hiveOtelAllLocal.systemd.services.hive-c0re.environment.HIVE_SQUID_UNRESTRICTED_FILE == list;
|
||||
}
|
||||
{
|
||||
# The helper is the NCSA one, on the file hive-c0re renders. The file
|
||||
# exists, empty, before squid starts (the helper exits on a missing one),
|
||||
|
|
|
|||
|
|
@ -66,6 +66,38 @@ let
|
|||
config.services.hyperhive.network.exposeHostPorts;
|
||||
};
|
||||
|
||||
# The uplink is a public range. The private ranges squid must refuse are
|
||||
# extra subnets on the same link, so the proxy can reach a listener in each:
|
||||
# a name that resolves there is refused only by squid's own rules.
|
||||
privateNets = [
|
||||
{
|
||||
name = "priv10.t.local";
|
||||
net = "10.99.0";
|
||||
}
|
||||
{
|
||||
name = "priv172.t.local";
|
||||
net = "172.16.99";
|
||||
}
|
||||
{
|
||||
name = "priv192.t.local";
|
||||
net = "192.168.99";
|
||||
}
|
||||
{
|
||||
name = "cgnat.t.local";
|
||||
net = "100.64.99";
|
||||
}
|
||||
];
|
||||
privateAddrs =
|
||||
host:
|
||||
map (p: {
|
||||
address = "${p.net}.${toString host}";
|
||||
prefixLength = 24;
|
||||
}) privateNets;
|
||||
ulaAddr = host: {
|
||||
address = "fd00:99::${toString host}";
|
||||
prefixLength = 64;
|
||||
};
|
||||
|
||||
# Plain TCP on 443: a CONNECT is answered before any TLS, which is all the
|
||||
# test reads.
|
||||
listen443 = addrs: {
|
||||
|
|
@ -88,9 +120,10 @@ pkgs.testers.runNixOSTest {
|
|||
imports = [
|
||||
hiveStubs
|
||||
../host-modules/hive-squid.nix
|
||||
# Listeners a leaked destination would reach: loopback and the bridge.
|
||||
# Listeners a leaked destination would reach: loopback (`::` lands on `::1`) and the bridge.
|
||||
(listen443 [
|
||||
"127.0.0.1"
|
||||
"[::1]"
|
||||
bridgeIp
|
||||
])
|
||||
];
|
||||
|
|
@ -104,40 +137,73 @@ pkgs.testers.runNixOSTest {
|
|||
prefixLength = 24;
|
||||
}
|
||||
];
|
||||
networking.interfaces.eth2.ipv4.addresses = lib.mkForce [
|
||||
{
|
||||
address = "192.168.2.1";
|
||||
prefixLength = 24;
|
||||
}
|
||||
];
|
||||
networking.interfaces.eth2.ipv4.addresses = lib.mkForce (
|
||||
[
|
||||
{
|
||||
address = "203.0.113.1";
|
||||
prefixLength = 24;
|
||||
}
|
||||
]
|
||||
++ privateAddrs 1
|
||||
);
|
||||
networking.interfaces.eth2.ipv6.addresses = [ (ulaAddr 1) ];
|
||||
# Names the proxy resolves, each a different destination class.
|
||||
networking.hosts = {
|
||||
"192.168.2.2" = [ "forge.t.local" ];
|
||||
# `other.t.local` and the self-check's deny canary resolve, are
|
||||
# reachable and are on no allow-list. The canary is in the hosts file
|
||||
# because the destination denies look every name up first, and the VM
|
||||
# has no resolver to answer for an unknown one before the client gives up.
|
||||
"203.0.113.2" = [
|
||||
"forge.t.local"
|
||||
"other.t.local"
|
||||
"deny-canary.invalid"
|
||||
];
|
||||
"127.0.0.1" = [ "loop.t.local" ];
|
||||
${bridgeIp} = [ "bridge.t.local" ];
|
||||
"169.254.169.254" = [ "meta.t.local" ];
|
||||
};
|
||||
"fd00:99::2" = [ "ula.t.local" ];
|
||||
"::" = [ "any6.t.local" ];
|
||||
}
|
||||
// lib.listToAttrs (
|
||||
map (p: {
|
||||
name = "${p.net}.2";
|
||||
value = [ p.name ];
|
||||
}) privateNets
|
||||
);
|
||||
# Renders the users file the way hive-c0re does.
|
||||
environment.systemPackages = [ pkgs.mkpasswd ];
|
||||
services.hyperhive.squid.selfcheck.unrestrictedCanary = "other.t.local";
|
||||
services.hyperhive.squid.allowedDomains = [
|
||||
"forge.t.local"
|
||||
"loop.t.local"
|
||||
"bridge.t.local"
|
||||
"meta.t.local"
|
||||
];
|
||||
"ula.t.local"
|
||||
"any6.t.local"
|
||||
]
|
||||
++ map (p: p.name) privateNets;
|
||||
};
|
||||
|
||||
web =
|
||||
{ lib, ... }:
|
||||
{
|
||||
imports = [ (listen443 [ "0.0.0.0" ]) ];
|
||||
virtualisation.vlans = [ 2 ];
|
||||
networking.interfaces.eth1.ipv4.addresses = lib.mkForce [
|
||||
{
|
||||
address = "192.168.2.2";
|
||||
prefixLength = 24;
|
||||
}
|
||||
imports = [
|
||||
(listen443 [
|
||||
"0.0.0.0"
|
||||
"[::]"
|
||||
])
|
||||
];
|
||||
virtualisation.vlans = [ 2 ];
|
||||
networking.interfaces.eth1.ipv4.addresses = lib.mkForce (
|
||||
[
|
||||
{
|
||||
address = "203.0.113.2";
|
||||
prefixLength = 24;
|
||||
}
|
||||
]
|
||||
++ privateAddrs 2
|
||||
);
|
||||
networking.interfaces.eth1.ipv6.addresses = [ (ulaAddr 2) ];
|
||||
networking.firewall.allowedTCPPorts = [ 443 ];
|
||||
};
|
||||
|
||||
|
|
@ -157,14 +223,24 @@ pkgs.testers.runNixOSTest {
|
|||
testScript = ''
|
||||
import re
|
||||
import secrets
|
||||
import time
|
||||
|
||||
# Drawn fresh each run. The users file holds only their hashes, as
|
||||
# hive-c0re renders it.
|
||||
PW = {u: secrets.token_urlsafe(24) for u in ("atlas", "selfcheck.squid", "ruth")}
|
||||
# hive-c0re renders it. `atlas` is a restricted agent and `uma` an
|
||||
# unrestricted one; the two self-check users share one secret.
|
||||
PW = {
|
||||
u: secrets.token_urlsafe(24)
|
||||
for u in ("atlas", "uma", "selfcheck.squid", "ruth")
|
||||
}
|
||||
PW["selfcheck-unrestricted.squid"] = PW["selfcheck.squid"]
|
||||
ATLAS = "atlas:" + PW["atlas"]
|
||||
UMA = "uma:" + PW["uma"]
|
||||
SELFCHECK = "selfcheck.squid:" + PW["selfcheck.squid"]
|
||||
SELFCHECK_U = "selfcheck-unrestricted.squid:" + PW["selfcheck.squid"]
|
||||
RUTH = "ruth:" + PW["ruth"]
|
||||
USERS = "/var/lib/hive-squid/users"
|
||||
LIST = "/var/lib/hive-squid/unrestricted"
|
||||
UNRESTRICTED = ["uma", "selfcheck-unrestricted.squid"]
|
||||
|
||||
start_all()
|
||||
proxy.wait_for_unit("squid.service")
|
||||
|
|
@ -173,6 +249,8 @@ pkgs.testers.runNixOSTest {
|
|||
web.wait_for_unit("nginx.service")
|
||||
proxy.wait_for_open_port(3128, "${bridgeIp}")
|
||||
|
||||
PRIVATE = ["priv10.t.local", "priv172.t.local", "priv192.t.local", "cgnat.t.local", "ula.t.local", "any6.t.local"]
|
||||
|
||||
def connect(host, cred=None):
|
||||
# The status squid answered the CONNECT with; 000 when none came back.
|
||||
auth = f"-U '{cred}' " if cred else ""
|
||||
|
|
@ -194,17 +272,47 @@ pkgs.testers.runNixOSTest {
|
|||
f"&& chmod 640 {USERS}.new && mv {USERS}.new {USERS} && sleep 1"
|
||||
)
|
||||
|
||||
with subtest("squid starts on the empty users file and refuses every login"):
|
||||
def write_list(*names):
|
||||
# What hive-c0re does for the list. Squid reads it on a reload, which
|
||||
# squid-unrestricted-reload does when the file changes.
|
||||
body = "printf '%s\\n' " + " ".join(names) if names else ":"
|
||||
proxy.succeed(
|
||||
f"({body}) > {LIST}.new && chown hive-core:squid {LIST}.new "
|
||||
f"&& chmod 640 {LIST}.new && mv {LIST}.new {LIST}"
|
||||
)
|
||||
|
||||
def wait_status(host, cred, want):
|
||||
# The reload that follows a list write is asynchronous.
|
||||
got = None
|
||||
for _ in range(30):
|
||||
got = connect(host, cred)
|
||||
if got == want:
|
||||
return
|
||||
time.sleep(1)
|
||||
raise AssertionError(f"{host} as {cred.split(':')[0]}: wanted {want}, last got {got}")
|
||||
|
||||
with subtest("the private listeners answer the proxy itself, so a refusal below is squid's"):
|
||||
for addr in ["10.99.0.2", "172.16.99.2", "192.168.99.2", "100.64.99.2", "fd00:99::2"]:
|
||||
proxy.wait_until_succeeds(f"timeout 3 bash -c 'echo > /dev/tcp/{addr}/443'")
|
||||
assert connect("forge.t.local", None) == "407", "control: the public name is challenged, not refused"
|
||||
|
||||
with subtest("squid starts on the empty users file and list and refuses every login"):
|
||||
assert proxy.succeed(f"stat -c '%U:%G %a' {USERS}").strip() == "hive-core:squid 640"
|
||||
assert proxy.succeed(f"stat -c '%U:%G %a' {LIST}").strip() == "hive-core:squid 640"
|
||||
assert proxy.succeed("stat -c %a /var/lib/hive-squid").strip() == "2750"
|
||||
assert proxy.succeed(f"stat -c %s {USERS}").strip() == "0"
|
||||
assert proxy.succeed(f"stat -c %s {LIST}").strip() == "0"
|
||||
assert connect("forge.t.local", ATLAS) == "407", "no such user"
|
||||
assert connect("deny-canary.invalid", ATLAS) == "403", "an unlisted name needs no login"
|
||||
assert connect("other.t.local", UMA) == "407", "no such user, unlisted name"
|
||||
|
||||
with subtest("the users file holds hashes, never a password"):
|
||||
write_users(ATLAS, SELFCHECK)
|
||||
write_users(ATLAS, UMA, SELFCHECK, SELFCHECK_U)
|
||||
users = proxy.succeed(f"cat {USERS}")
|
||||
assert re.fullmatch(r"(atlas|selfcheck\.squid):\$6\$\S+\n(atlas|selfcheck\.squid):\$6\$\S+\n", users), users
|
||||
rows = [l.split(":", 1) for l in users.splitlines()]
|
||||
assert sorted(r[0] for r in rows) == sorted(
|
||||
["atlas", "uma", "selfcheck.squid", "selfcheck-unrestricted.squid"]
|
||||
), users
|
||||
assert all(re.fullmatch(r"\$6\$\S+", r[1]) for r in rows), users
|
||||
assert not any(p in users for p in PW.values()), users
|
||||
|
||||
with subtest("an allowed name is tunnelled with a credential and challenged without one"):
|
||||
|
|
@ -215,22 +323,33 @@ pkgs.testers.runNixOSTest {
|
|||
# DES would accept the first eight characters; the hash must not.
|
||||
assert connect("forge.t.local", "atlas:" + PW["atlas"][:8]) == "407", "a prefix of the password"
|
||||
|
||||
with subtest("an unlisted name is refused with or without a credential"):
|
||||
assert connect("deny-canary.invalid", ATLAS) == "403", "deny canary"
|
||||
assert connect("deny-canary.invalid") == "403", "deny canary, anonymous"
|
||||
with subtest("with an empty list every user is restricted"):
|
||||
assert connect("other.t.local", UMA) == "403", "uma, not listed yet"
|
||||
assert connect("other.t.local", ATLAS) == "403"
|
||||
assert connect("forge.t.local", UMA) == "200", "control: uma still reaches a listed name"
|
||||
|
||||
with subtest("a name resolving to a host-local destination is refused"):
|
||||
assert connect("loop.t.local", ATLAS) == "403", "loopback"
|
||||
assert connect("bridge.t.local", ATLAS) == "403", "bridge address"
|
||||
assert connect("meta.t.local", ATLAS) == "403", "link-local"
|
||||
with subtest("a listed user reaches an unlisted name and the others are refused"):
|
||||
write_list(*UNRESTRICTED)
|
||||
wait_status("other.t.local", UMA, "200")
|
||||
assert connect("other.t.local", ATLAS) == "403", "restricted"
|
||||
assert connect("forge.t.local", ATLAS) == "200", "restricted, listed name"
|
||||
assert connect("forge.t.local", UMA) == "200", "unrestricted, listed name"
|
||||
assert connect("other.t.local") == "407", "no credential"
|
||||
assert connect("other.t.local", "uma:" + secrets.token_urlsafe(24)) == "407", "wrong password"
|
||||
assert connect("deny-canary.invalid", ATLAS) == "403", "restricted, an unlisted name"
|
||||
|
||||
with subtest("IP literals are refused even when the address is reachable"):
|
||||
assert connect("127.0.0.1", ATLAS) == "403"
|
||||
assert connect("192.168.2.2", ATLAS) == "403"
|
||||
|
||||
with subtest("only a port-443 CONNECT is allowed"):
|
||||
assert connect("forge.t.local:22", ATLAS) == "403"
|
||||
assert connect("forge.t.local:80", ATLAS) == "403"
|
||||
with subtest("neither kind of user reaches a host-local or private destination, or another port"):
|
||||
for cred in (ATLAS, UMA):
|
||||
who = cred.split(":")[0]
|
||||
assert connect("loop.t.local", cred) == "403", f"{who}: loopback"
|
||||
assert connect("bridge.t.local", cred) == "403", f"{who}: bridge address"
|
||||
assert connect("meta.t.local", cred) == "403", f"{who}: link-local"
|
||||
assert connect("127.0.0.1", cred) == "403", f"{who}: loopback literal"
|
||||
assert connect("203.0.113.2", cred) == "403", f"{who}: IP literal"
|
||||
for name in PRIVATE:
|
||||
assert connect(name, cred) == "403", f"{who}: {name}, a listed name that resolves to a private address"
|
||||
assert connect("other.t.local:22", cred) == "403", f"{who}: port 22"
|
||||
assert connect("other.t.local:80", cred) == "403", f"{who}: CONNECT to port 80"
|
||||
|
||||
with subtest("an agent cannot read cachemgr"):
|
||||
agent.succeed(
|
||||
|
|
@ -247,32 +366,50 @@ pkgs.testers.runNixOSTest {
|
|||
"journalctl -u squid-selfcheck.service -o cat | "
|
||||
"grep -F 'squid-selfcheck ok=true allow_canary=forge.t.local "
|
||||
"allow_status=200 deny_canary=deny-canary.invalid deny_status=403 "
|
||||
"anon_status=407 credential=present'"
|
||||
"anon_status=407 credential=present "
|
||||
"unrestricted_canary=other.t.local unrestricted_status=200'"
|
||||
)
|
||||
|
||||
with subtest("the self-check fails, and says why, when the unrestricted user is not listed"):
|
||||
write_list()
|
||||
wait_status("other.t.local", UMA, "403")
|
||||
proxy.fail("systemctl start squid-selfcheck.service")
|
||||
proxy.succeed(
|
||||
"journalctl -u squid-selfcheck.service -o cat | "
|
||||
"grep -F 'squid-selfcheck ok=false allow_canary=forge.t.local "
|
||||
"allow_status=200 deny_canary=deny-canary.invalid deny_status=403 "
|
||||
"anon_status=407 credential=present "
|
||||
"unrestricted_canary=other.t.local unrestricted_status=403'"
|
||||
)
|
||||
write_list(*UNRESTRICTED)
|
||||
wait_status("other.t.local", UMA, "200")
|
||||
|
||||
with subtest("the self-check fails, and says why, without a credential or with the wrong one"):
|
||||
proxy.succeed("mv /run/credstore/hive-squid.selfcheck /run/credstore/kept")
|
||||
proxy.fail("systemctl start squid-selfcheck.service")
|
||||
proxy.succeed(
|
||||
"journalctl -u squid-selfcheck.service -o cat | "
|
||||
"grep -F 'squid-selfcheck ok=false allow_canary=forge.t.local "
|
||||
"allow_status=407 deny_canary=deny-canary.invalid deny_status=403 "
|
||||
"anon_status=407 credential=absent'"
|
||||
"allow_status=407 deny_canary=deny-canary.invalid deny_status=407 "
|
||||
"anon_status=407 credential=absent "
|
||||
"unrestricted_canary=other.t.local unrestricted_status=407'"
|
||||
)
|
||||
proxy.succeed(f"printf '%s\\n' {secrets.token_urlsafe(24)} > /run/credstore/hive-squid.selfcheck")
|
||||
proxy.fail("systemctl start squid-selfcheck.service")
|
||||
proxy.succeed(
|
||||
"journalctl -u squid-selfcheck.service -o cat | "
|
||||
"grep -F 'allow_status=407 deny_canary=deny-canary.invalid deny_status=403 "
|
||||
"anon_status=407 credential=present'"
|
||||
"grep -F 'allow_status=407 deny_canary=deny-canary.invalid deny_status=407 "
|
||||
"anon_status=407 credential=present "
|
||||
"unrestricted_canary=other.t.local unrestricted_status=407'"
|
||||
)
|
||||
proxy.succeed("mv -f /run/credstore/kept /run/credstore/hive-squid.selfcheck")
|
||||
|
||||
with subtest("squid's access log reaches the journal under squid.service, with the user"):
|
||||
journal = proxy.succeed("journalctl -u squid.service -o cat")
|
||||
assert re.search(r"192\.168\.1\.3 TCP_TUNNEL/200 .* CONNECT forge\.t\.local:443 atlas ", journal), journal
|
||||
assert re.search(r"192\.168\.1\.3 TCP_TUNNEL/200 .* CONNECT other\.t\.local:443 uma ", journal), journal
|
||||
assert re.search(r"192\.168\.1\.3 TCP_DENIED/407 .* CONNECT forge\.t\.local:443 - ", journal), journal
|
||||
assert re.search(r"192\.168\.1\.3 TCP_DENIED/403 .* CONNECT deny-canary\.invalid:443 ", journal), journal
|
||||
assert re.search(r"192\.168\.1\.3 TCP_DENIED/403 .* CONNECT other\.t\.local:443 atlas ", journal), journal
|
||||
assert re.search(r"192\.168\.1\.3 TCP_DENIED/403 .* CONNECT loop\.t\.local:443 ", journal), journal
|
||||
# Exporter scrapes are cachemgr requests and stay out of the log.
|
||||
assert "squid-internal-mgr" not in journal, journal
|
||||
|
|
@ -287,15 +424,36 @@ pkgs.testers.runNixOSTest {
|
|||
assert m and float(m.group(1)) >= 1, metrics
|
||||
proxy.fail("curl -sf -m 5 http://${bridgeIp}:9301/metrics")
|
||||
|
||||
with subtest("squid survives a reload"):
|
||||
with subtest("squid survives a reload and keeps the list"):
|
||||
proxy.succeed("systemctl reload squid.service")
|
||||
proxy.wait_for_unit("squid.service")
|
||||
assert connect("forge.t.local", ATLAS) == "200"
|
||||
assert connect("other.t.local", UMA) == "200"
|
||||
assert connect("other.t.local", ATLAS) == "403"
|
||||
|
||||
with subtest("a user added while squid runs can log in, with no reload"):
|
||||
write_users(ATLAS, SELFCHECK, RUTH)
|
||||
with subtest("a user added while squid runs can log in, with no reload, and is restricted"):
|
||||
write_users(ATLAS, UMA, SELFCHECK, SELFCHECK_U, RUTH)
|
||||
assert connect("forge.t.local", RUTH) == "200"
|
||||
assert connect("forge.t.local", ATLAS) == "200"
|
||||
assert connect("other.t.local", RUTH) == "403"
|
||||
|
||||
with subtest("a user the list gains is unrestricted after the reload and loses it again"):
|
||||
write_list(*UNRESTRICTED, "ruth")
|
||||
wait_status("other.t.local", RUTH, "200")
|
||||
assert connect("loop.t.local", RUTH) == "403", "still no loopback"
|
||||
write_list(*UNRESTRICTED)
|
||||
wait_status("other.t.local", RUTH, "403")
|
||||
assert connect("other.t.local", UMA) == "200", "control: uma is still listed"
|
||||
|
||||
with subtest("squid starts on a missing list, treats every user as restricted, and recovers when it appears"):
|
||||
proxy.succeed(f"rm {LIST}")
|
||||
proxy.succeed("systemctl restart squid.service")
|
||||
proxy.wait_for_unit("squid.service")
|
||||
proxy.wait_for_open_port(3128, "${bridgeIp}")
|
||||
assert connect("forge.t.local", UMA) == "200", "logins work"
|
||||
assert connect("other.t.local", UMA) == "403", "no list, no unrestricted user"
|
||||
write_list(*UNRESTRICTED)
|
||||
wait_status("other.t.local", UMA, "200")
|
||||
|
||||
with subtest("squid starts on a missing users file, refuses logins, and recovers when it appears"):
|
||||
proxy.succeed(f"rm {USERS}")
|
||||
|
|
@ -303,8 +461,9 @@ pkgs.testers.runNixOSTest {
|
|||
proxy.wait_for_unit("squid.service")
|
||||
proxy.wait_for_open_port(3128, "${bridgeIp}")
|
||||
assert connect("forge.t.local", ATLAS) == "407", "no file, no login"
|
||||
assert connect("deny-canary.invalid", ATLAS) == "403", "an unlisted name still answers"
|
||||
write_users(ATLAS, SELFCHECK)
|
||||
assert connect("other.t.local", UMA) == "407", "no file, no login, even when listed"
|
||||
write_users(ATLAS, UMA, SELFCHECK, SELFCHECK_U)
|
||||
assert connect("forge.t.local", ATLAS) == "200", "the file appeared"
|
||||
assert connect("other.t.local", UMA) == "200", "listed, and now logged in"
|
||||
'';
|
||||
}
|
||||
|
|
|
|||
|
|
@ -28,6 +28,14 @@ pub const SELFCHECK_LEAF: &str = "squid-selfcheck";
|
|||
/// `nix/host-modules/hive-squid.nix` spells the same string.
|
||||
pub const SELFCHECK_USER: &str = "selfcheck.squid";
|
||||
|
||||
/// The squid user the self-check logs in as to prove an unrestricted agent
|
||||
/// reaches an unlisted name. It authenticates with the same secret as
|
||||
/// [`SELFCHECK_USER`] ([`selfcheck_path`]), and is always on squid's
|
||||
/// unrestricted list. It cannot be an agent's name for the same reason as
|
||||
/// [`SELFCHECK_USER`]. `nix/host-modules/hive-squid.nix` spells the same
|
||||
/// string.
|
||||
pub const SELFCHECK_UNRESTRICTED_USER: &str = "selfcheck-unrestricted.squid";
|
||||
|
||||
/// The path holding the secret `agent` authenticates to the egress proxy with.
|
||||
///
|
||||
/// A flat leaf under the agent's prefix, like
|
||||
|
|
@ -127,8 +135,13 @@ mod tests {
|
|||
}
|
||||
|
||||
#[test]
|
||||
fn the_selfcheck_user_is_not_a_legal_agent_name() {
|
||||
assert!(crate::path::checked_segment("agent", SELFCHECK_USER).is_err());
|
||||
fn the_selfcheck_users_are_not_legal_agent_names() {
|
||||
for user in [SELFCHECK_USER, SELFCHECK_UNRESTRICTED_USER] {
|
||||
assert!(
|
||||
crate::path::checked_segment("agent", user).is_err(),
|
||||
"{user}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
|
|
|||
Loading…
Reference in a new issue