Watch
0
0
Fork
You've already forked hyperhive
0

Compare commits

..
10 changed files with 162 additions and 827 deletions

View file

@ -230,46 +230,18 @@ host. It never decrypts TLS, so it holds no signing key. It caches nothing.
store and the next pass mints a new one. HTTP Basic crosses the bridge in
clear between an agent and the proxy. See
[the credentials reference](../swarm/credentials-reference.md).
A request without a valid credential gets a 407, once the destination
checks below have passed it.
- **Two lists.** The credential decides the policy. `hive-c0re` writes a
second file beside the users file, `services.hyperhive.squid.unrestrictedFile`
(`/var/lib/hive-squid/unrestricted`, `hive-core:squid`, `0640`): the names of
the users whose agent holds the `unrestricted_internet` flag in the
permissions the hive last read from the swarm, one per line, rewritten when
it changes. The default is permissive: an agent with no record, or a record
that never set the flag, holds it, the same default as the
`hyperhive.agent.unrestricted_internet` metric. While the hive's read of the
permissions is stale or failing, a new agent that already holds a
credential counts as unrestricted. Squid reads the list when it
parses its configuration, so `squid-unrestricted-reload.path` reloads squid
when the file changes. A missing or empty file lists no one, and squid
starts either way. A flag change reaches squid after the hive's next read of
the permissions and its next 30-second pass.
An older `hive-c0re` writes no list, so every credential on that hive is
restricted. Agents must not start sending credentials until every hive runs
the `hive-c0re` that writes it, or a legacy agent loses general internet.
A name that isn't on the allow-list gets the 403 whether or not the request
has a credential; a listed name without one gets a 407.
- **Allow-list.** `services.hyperhive.squid.allowedDomains` defaults to the
swarm's own service names (`services.hyperhive.swarm.serviceDomains`). Each
entry matches that exact name. A user on the unrestricted list reaches any
other name too, on ports 80 and 443. Everyone else gets a 403 for it.
- **Unlisted names.** For a user who isn't on the unrestricted list, refused
after the destination checks, which look the name up on the host.
entry matches that exact name. Everything else gets a 403.
- **Unlisted names.** Refused first, without a DNS lookup.
- **Denied before any allow.** Destinations that resolve to loopback,
link-local, the bridge subnet, a private range (`10.0.0.0/8`,
`172.16.0.0/12`, `192.168.0.0/16`, `fc00::/7`), shared address space
(`100.64.0.0/10`), `0.0.0.0/8`, `::`, multicast or reserved space, and IP-literal
hosts, for every user including the unrestricted ones and for listed names.
Squid resolves names on the host and connects from it, so without these a
name pointing at `127.0.0.1`, at the host's own LAN address or at a private
neighbour would walk around the bridge→loopback DROP rule and the swarm's
network. Squid checks the address it connects to, so a name that changes its
answer between a check and the connect gains nothing. A listed name must
therefore resolve to a public address on the host's resolver: a split-horizon
name that answers a LAN address gets a 403, and so does the self-check
canary. Only `CONNECT` to port 443 and plain HTTP to ports 80 and 443 pass.
link-local or the bridge subnet, and IP-literal hosts. Squid resolves names
on the host, so without these a listed name pointing at `127.0.0.1` would
walk around the bridge→loopback DROP rule. Only `CONNECT` to port 443 and
plain HTTP to ports 80 and 443 pass.
- **Metrics.** `squid-exporter` serves `prometheus-squid-exporter` on
`127.0.0.1:9301`. The hive collector scrapes it as job `squid` through
`services.hyperhive.otel.scrapeTargets`. `squid_up` and
@ -282,13 +254,9 @@ host. It never decrypts TLS, so it holds no signing key. It caches nothing.
every minute from the bridge IP, logging in as `selfcheck.squid`: `CONNECT`
to `services.hyperhive.squid.selfcheck.canary` (default: the forge's name)
must answer 200, to `deny-canary.invalid` 403, and to the canary without a
credential 407. It then logs in as `selfcheck-unrestricted.squid`, which has
the same secret and is always on the unrestricted list, and a `CONNECT` to
`services.hyperhive.squid.selfcheck.unrestrictedCanary` (default:
`example.com`, a name outside the allow-list) must answer 200. It writes one
`squid-selfcheck ok=… allow_status=… deny_status=… anon_status=…
credential=present|absent unrestricted_canary=… unrestricted_status=…` line
per run and fails the unit when any is wrong. The password is the hive's
credential 407. It writes one `squid-selfcheck ok=… allow_status=…
deny_status=… anon_status=… credential=present|absent` line per run and
fails the unit when any is wrong. The password is the hive's
`swarm/hives/<hive>/squid-selfcheck` secret, which
`squid-selfcheck-credential` lands at `/run/credstore/hive-squid.selfcheck`.
Before `swarm-controller` has minted it, or while `hive-c0re` hasn't

View file

@ -320,10 +320,7 @@ an absent flag included, and a value in the request wins. A `PUT` that leaves
the field out and a `DELETE` (`swarmctl agent permissions clear`) also keep the
stored value; a `DELETE` on an agent that has one rewrites the record to just
that value. Read it with
`swarm_queue_client::agent_permissions::unrestricted`. The hive reads it for
the agent's squid policy: [the egress proxy](../networking/network.md#egress-proxy-squid)
lets an unrestricted agent's credential reach any destination, and holds the
others to the allow-list. No agent sends a credential yet.
`swarm_queue_client::agent_permissions::unrestricted`. Nothing acts on it yet.
**How a change applies** —
- The agent's harness reads its own key every 10 seconds and builds every

View file

@ -148,16 +148,16 @@ pieces fit, and the reasoning behind them, is in [`credentials.md`](credentials.
### Hive-level — one of each per hive
| secret | generated by | lives at |
| ---------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` |
| the cache's public key and pull token, for hive-priv | `swarm-controller` copies them to `swarm/hives/<hive>/attic/{public-key,pull-token}` (see the cache-facts row above) | `hive-priv-cache-facts` (root) reads those two under the hive's own store identity every ten minutes and writes `/run/hyperhive-priv/attic/public-key` and `netrc` (tmpfs, root, each `0600`, directory `0700`); the cache's address comes from `services.hyperhive.swarm.attic.domain` and `cache`, not from the store. A store that holds no facts, or that doesn't answer, leaves the files as they stand, and a hive that never got them has `hive-priv` answer `not available`. |
| the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/run/hive-forge-oidc/<id>.secret` on the host (tmpfs), bound into the forge container, handed to `forgejo-sso-source` through `LoadCredential` |
| the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/<id>.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` |
| this hive's forge read token | `swarm-controller`, into the secret store at `swarm/hives/<hive>/forge/token`, for the `hive-<hive>` account; a pass at start and every five minutes re-mints a missing or stale token | `hive-c0re` reads it under the hive's own store identity, and copies it (account on the first line, token on the second) to `/run/hyperhive/forge-hive-token` (tmpfs, `0600` `hive-core`) for nix's `git`, which `hive-priv` also runs. Each `internal/meta` read and each boot rewrites the copy when it differs; where they differ after a rotation, a build that starts before the next read fails to authenticate, and a retry reads the new token |
| squid's users file | `hive-c0re` (`workers::squid_users`), every 30 seconds, from the proxy credentials of its agents and the self-check | `services.hyperhive.squid.usersFile` (`/var/lib/hive-squid/users`, `hive-core:squid`, `0640`): `<user>:<hash>` lines, SHA-512 crypt hashes only, renamed into place when it changed. `unrestricted` beside it lists the users whose agent holds the swarm's `unrestricted_internet` flag, names only; it lists every agent the hive never read as off `systemd-tmpfiles` creates it empty so squid's helper never sees a missing file. A hive whose agents have no credential yet (an older controller) has an empty file: every login fails and the self-check reports `ok=false`; no agent sends a credential, so no traffic changes |
| the squid self-check's password | `swarm-controller` mints `swarm/hives/<hive>/squid-selfcheck`; `squid-selfcheck-credential` (root, every 10 minutes) reads it as the hive | `/run/credstore/hive-squid.selfcheck` (tmpfs, root, `0400`), imported by `squid-selfcheck.service` as a systemd credential and handed to `curl` on its stdin. Both self-check users (`selfcheck.squid` and the always-unrestricted `selfcheck-unrestricted.squid`) use it, so whoever holds it has unrestricted egress through this squid Missing, or different from the users file: the self-check logs `credential=absent` or a 407 and fails, and nothing else breaks |
| secret | generated by | lives at |
| ---------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` |
| the cache's public key and pull token, for hive-priv | `swarm-controller` copies them to `swarm/hives/<hive>/attic/{public-key,pull-token}` (see the cache-facts row above) | `hive-priv-cache-facts` (root) reads those two under the hive's own store identity every ten minutes and writes `/run/hyperhive-priv/attic/public-key` and `netrc` (tmpfs, root, each `0600`, directory `0700`); the cache's address comes from `services.hyperhive.swarm.attic.domain` and `cache`, not from the store. A store that holds no facts, or that doesn't answer, leaves the files as they stand, and a hive that never got them has `hive-priv` answer `not available`. |
| the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/run/hive-forge-oidc/<id>.secret` on the host (tmpfs), bound into the forge container, handed to `forgejo-sso-source` through `LoadCredential` |
| the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/<id>.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` |
| this hive's forge read token | `swarm-controller`, into the secret store at `swarm/hives/<hive>/forge/token`, for the `hive-<hive>` account; a pass at start and every five minutes re-mints a missing or stale token | `hive-c0re` reads it under the hive's own store identity, and copies it (account on the first line, token on the second) to `/run/hyperhive/forge-hive-token` (tmpfs, `0600` `hive-core`) for nix's `git`, which `hive-priv` also runs. Each `internal/meta` read and each boot rewrites the copy when it differs; where they differ after a rotation, a build that starts before the next read fails to authenticate, and a retry reads the new token |
| squid's users file | `hive-c0re` (`workers::squid_users`), every 30 seconds, from the proxy credentials of its agents and the self-check | `services.hyperhive.squid.usersFile` (`/var/lib/hive-squid/users`, `hive-core:squid`, `0640`): `<user>:<hash>` lines, SHA-512 crypt hashes only, renamed into place when it changed. `systemd-tmpfiles` creates it empty so squid's helper never sees a missing file. A hive whose agents have no credential yet (an older controller) has an empty file: every login fails and the self-check reports `ok=false`; no agent sends a credential, so no traffic changes |
| the squid self-check's password | `swarm-controller` mints `swarm/hives/<hive>/squid-selfcheck`; `squid-selfcheck-credential` (root, every 10 minutes) reads it as the hive | `/run/credstore/hive-squid.selfcheck` (tmpfs, root, `0400`), imported by `squid-selfcheck.service` as a systemd credential and handed to `curl` on its stdin. Missing, or different from the users file: the self-check logs `credential=absent` or a 407 and fails, and nothing else breaks |
### Store identities, one per principal

View file

@ -568,9 +568,7 @@ the swarm never set the flag, which counts as on. The **restrict** or
**allow unrestricted** badge flips it with a `PUT` that carries the stored
permission list along with the flag, because a `PUT` without the list resets a
stored list to the default. An agent with no stored list sends the flag alone.
The hive's egress proxy applies the flag to each agent's proxy credential (see
[the egress proxy](../networking/network.md#egress-proxy-squid)); no agent sends
one yet, so the flag changes no traffic today.
Nothing acts on the flag yet.
A saved change isn't instant. The hive reads the caps about every 30 seconds,
and a running agent takes a changed cap at its next spawn or rebuild, so the

View file

@ -1,6 +1,5 @@
//! Squid's users file, and its list of unrestricted users: the hashes of the
//! proxy credentials this hive's agents authenticate to its egress proxy with,
//! and which of them may reach any destination.
//! Squid's users file: the hashes of the proxy credentials this hive's agents
//! authenticate to its egress proxy with.
//!
//! swarm-controller mints one credential per agent and one for this hive's own
//! squid self-check ([`swarm_secret_client::proxy`]). Every [`INTERVAL`] this
@ -10,19 +9,12 @@
//! moves. The agents are the container list, not the running containers: a
//! stopped agent stays in the file.
//!
//! The file named by [`ENV_UNRESTRICTED_FILE`] lists, one name per line, the
//! users whose agent holds `unrestricted_internet` in the permissions this
//! hive last read from the swarm (an agent with no record, or no flag in it,
//! holds it), plus the self-check's unrestricted user. Squid reads the list as
//! an ACL when it parses its configuration, so `nix/host-modules/hive-squid.nix`
//! reloads squid when the file changes.
//!
//! Only salted SHA-512 crypt hashes reach the disk. A hash is kept as it is
//! while it still verifies its credential, so each file changes only when a
//! user is added, removed, given a new credential or has its flag flipped.
//! while it still verifies its credential, so the file changes only when a
//! user is added, removed or given a new credential.
//!
//! A failed read leaves both files as they are: dropping a user on a store
//! outage would read exactly like the swarm revoking it.
//! A failed read leaves the file as it is: dropping a user on a store outage
//! would read exactly like the swarm revoking it.
use std::future::Future;
use std::path::{Path, PathBuf};
@ -32,7 +24,6 @@ use anyhow::{Context, Result, bail};
use sha_crypt::{PasswordHasher as _, PasswordVerifier as _, ShaCrypt};
use swarm_secret_client::{SecretStore, client, proxy};
use crate::permissions;
use crate::stats::sweep_health::{self, SweepHealth};
/// How often the file is re-rendered.
@ -45,15 +36,11 @@ const FAILURES_BEFORE_WARNING: u32 = 3;
/// from `services.hyperhive.squid.usersFile`; unset means this host runs no squid.
pub const ENV_USERS_FILE: &str = "HIVE_SQUID_USERS_FILE";
/// The environment variable naming the unrestricted-users list. Set by
/// `nix/host-modules/hive-c0re` from `services.hyperhive.squid.unrestrictedFile`.
pub const ENV_UNRESTRICTED_FILE: &str = "HIVE_SQUID_UNRESTRICTED_FILE";
/// Raw salt bytes: twelve of them encode to the sixteen characters `crypt()`
/// accepts.
const SALT_BYTES: usize = 12;
/// Both files' mode. The group is squid's, from the setgid directory
/// The file's mode. The group is squid's, from the setgid directory
/// `nix/host-modules/hive-squid.nix` creates.
const FILE_MODE: u32 = 0o640;
@ -62,32 +49,14 @@ const FILE_MODE: u32 = 0o640;
struct User {
name: String,
secret: String,
/// Whether squid lets this user reach any destination the deny rules leave.
unrestricted: bool,
}
/// The two files a pass writes.
struct Files<'a> {
users: &'a Path,
unrestricted: &'a Path,
}
/// Whether `name` is one of the self-check's squid users rather than an agent's.
fn is_selfcheck(name: &str) -> bool {
name == proxy::SELFCHECK_USER || name == proxy::SELFCHECK_UNRESTRICTED_USER
}
fn env_path(name: &str) -> Option<PathBuf> {
std::env::var_os(name)
.filter(|v| !v.is_empty())
.map(PathBuf::from)
}
/// Start the pass loop, when this host has squid's two files, a store and a
/// Start the pass loop, when this host has a squid users file, a store and a
/// hive name.
pub fn spawn(mut shutdown: tokio::sync::watch::Receiver<bool>) {
let (Some(users), Some(unrestricted)) =
(env_path(ENV_USERS_FILE), env_path(ENV_UNRESTRICTED_FILE))
let Some(path) = std::env::var_os(ENV_USERS_FILE)
.filter(|v| !v.is_empty())
.map(PathBuf::from)
else {
return;
};
@ -104,15 +73,7 @@ pub fn spawn(mut shutdown: tokio::sync::watch::Receiver<bool>) {
tokio::spawn(async move {
let mut health = SweepHealth::new("squid_users_render", "warn", FAILURES_BEFORE_WARNING);
loop {
match pass(
&hive,
&Files {
users: &users,
unrestricted: &unrestricted,
},
)
.await
{
match pass(&hive, &path).await {
Ok(()) => health.record_ok(),
Err(e) => {
tracing::warn!(error = ?e, "squid users: the pass failed");
@ -138,21 +99,18 @@ pub fn spawn(mut shutdown: tokio::sync::watch::Receiver<bool>) {
});
}
/// Read this hive's credentials from the store and the swarm's permissions
/// from the last read of them, and render both files.
async fn pass(hive: &str, files: &Files<'_>) -> Result<()> {
/// Read this hive's credentials from the store and render them into `file`.
async fn pass(hive: &str, file: &Path) -> Result<()> {
let agents: Vec<String> = crate::lifecycle::agents_for_meta_listing()
.await
.context("enumerating this hive's agents")?
.into_iter()
.map(|spec| spec.name)
.collect();
let perms = permissions::read_at(&permissions::path())
.context("reading the stored swarm permissions")?;
let store = SecretStore::from_env(hive)
.await
.context("connecting to the store")?;
sync_with(hive, &agents, &perms, files, |path| {
sync_with(hive, &agents, file, |path| {
let store = &store;
async move {
let held: Option<proxy::Credential> = store
@ -165,102 +123,47 @@ async fn pass(hive: &str, files: &Files<'_>) -> Result<()> {
.await
}
/// [`pass`] with the store read, the permissions and the files given. Nothing
/// is written unless every read succeeds, and a file is only written when its
/// rendering differs from what it holds.
///
/// The unrestricted list goes first. A user that turns restricted is then
/// restricted before the users file stops listing it, and one that turns
/// unrestricted is not listed before it can log in.
async fn sync_with<F, Fut>(
hive: &str,
agents: &[String],
perms: &permissions::Stored,
files: &Files<'_>,
read: F,
) -> Result<()>
/// [`pass`] with the store read and the file given. Nothing is written unless
/// every read succeeds and the rendered file differs from the current one.
async fn sync_with<F, Fut>(hive: &str, agents: &[String], file: &Path, read: F) -> Result<()>
where
F: Fn(String) -> Fut,
Fut: Future<Output = Result<Option<String>>>,
{
let mut wanted = Vec::new();
for agent in agents {
let unrestricted = permissions::unrestricted_internet(perms.get(agent));
wanted.push((
agent.clone(),
proxy::agent_credential_path(agent)?,
unrestricted,
));
wanted.push((agent.clone(), proxy::agent_credential_path(agent)?));
}
let selfcheck = proxy::selfcheck_path(hive)?;
wanted.push((proxy::SELFCHECK_USER.to_owned(), selfcheck.clone(), false));
wanted.push((
proxy::SELFCHECK_UNRESTRICTED_USER.to_owned(),
selfcheck,
true,
proxy::SELFCHECK_USER.to_owned(),
proxy::selfcheck_path(hive)?,
));
let mut users = Vec::new();
for (name, path, unrestricted) in wanted {
for (name, path) in wanted {
if let Some(secret) = read(path).await? {
users.push(User {
name,
secret,
unrestricted,
});
users.push(User { name, secret });
} else {
tracing::debug!(name, "squid users: no proxy credential minted yet");
}
}
let list = render_unrestricted(&users);
if write_if_changed(files.unrestricted, &list)? {
tracing::info!(
unrestricted = list.lines().count(),
file = %files.unrestricted.display(),
"squid users: wrote the unrestricted list"
);
}
let current = read_or_empty(files.users)?;
let current = match std::fs::read_to_string(file) {
Ok(text) => text,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => String::new(),
Err(e) => return Err(e).with_context(|| format!("reading {}", file.display())),
};
let rendered = render(&current, &users)?;
if write_if_changed(files.users, &rendered)? {
tracing::info!(
users = users.len(),
agents = users.iter().filter(|u| !is_selfcheck(&u.name)).count(),
file = %files.users.display(),
"squid users: wrote the users file"
);
if rendered == current {
return Ok(());
}
Ok(())
}
/// The file's text, or the empty string when it does not exist.
fn read_or_empty(file: &Path) -> Result<String> {
match std::fs::read_to_string(file) {
Ok(text) => Ok(text),
Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(String::new()),
Err(e) => Err(e).with_context(|| format!("reading {}", file.display())),
}
}
/// Write `text` to `file` unless it already holds exactly that. Returns
/// whether it wrote.
fn write_if_changed(file: &Path, text: &str) -> Result<bool> {
if read_or_empty(file)? == text {
return Ok(false);
}
crate::secret_file::write_secret_mode(file, text.as_bytes(), FILE_MODE)
crate::secret_file::write_secret_mode(file, rendered.as_bytes(), FILE_MODE)
.with_context(|| format!("writing {}", file.display()))?;
Ok(true)
}
/// The names of the unrestricted `users`, sorted, one per line.
fn render_unrestricted(users: &[User]) -> String {
let mut names: Vec<&str> = users
.iter()
.filter(|u| u.unrestricted)
.map(|u| u.name.as_str())
.collect();
names.sort_unstable();
names.iter().flat_map(|n| [n, "\n"]).collect()
tracing::info!(
users = users.len(),
agents = users.iter().filter(|u| u.name != proxy::SELFCHECK_USER).count(),
file = %file.display(),
"squid users: wrote the users file"
);
Ok(())
}
/// The users file for `users`, sorted by name, one `<user>:<hash>` line each.
@ -312,7 +215,6 @@ mod tests {
use std::os::unix::fs::PermissionsExt as _;
use super::*;
use swarm_wire::agent_permissions::AgentPermissions;
const HIVE: &str = "h1";
@ -328,54 +230,14 @@ mod tests {
names.iter().map(|n| (*n).to_owned()).collect()
}
/// The unrestricted list that sits beside the users file `users`.
fn list_of(users: &Path) -> PathBuf {
users.with_file_name("unrestricted")
}
/// The permissions record of an agent whose swarm flag is `flag`.
fn flagged(agent: &str, flag: Option<bool>) -> (String, AgentPermissions) {
let json = match flag {
Some(f) => format!(r#"{{"unrestricted_internet":{f}}}"#),
None => "{}".to_owned(),
};
(
agent.to_owned(),
serde_json::from_str(&json).expect("decodes"),
)
}
async fn sync_flags(
held: &HashMap<String, String>,
agents: &[&str],
perms: &permissions::Stored,
file: &Path,
) -> Result<()> {
let list = list_of(file);
let files = Files {
users: file,
unrestricted: &list,
};
sync_with(HIVE, &names(agents), perms, &files, |path| {
async fn sync(held: &HashMap<String, String>, agents: &[&str], file: &Path) -> Result<()> {
sync_with(HIVE, &names(agents), file, |path| {
let found = held.get(&path).cloned();
async move { Ok(found) }
})
.await
}
/// [`sync_flags`] with no permissions read yet: every agent unrestricted.
async fn sync(held: &HashMap<String, String>, agents: &[&str], file: &Path) -> Result<()> {
sync_flags(held, agents, &permissions::Stored::new(), file).await
}
fn list(users: &Path) -> Vec<String> {
std::fs::read_to_string(list_of(users))
.expect("the list is there")
.lines()
.map(str::to_owned)
.collect()
}
fn lines(file: &Path) -> Vec<(String, String)> {
std::fs::read_to_string(file)
.expect("the file is there")
@ -405,15 +267,7 @@ mod tests {
.await
.expect("renders");
let users: Vec<String> = lines(&file).into_iter().map(|(n, _)| n).collect();
assert_eq!(
users,
[
"argus",
"atlas",
"selfcheck-unrestricted.squid",
"selfcheck.squid"
]
);
assert_eq!(users, ["argus", "atlas", "selfcheck.squid"]);
}
#[tokio::test]
@ -444,17 +298,17 @@ mod tests {
sync(&two_agents_and_a_stranger(), &["atlas", "argus"], &file)
.await
.expect("renders");
// Control: the search finds a name that is in each file.
// Control: the search finds a name that is in the file.
let mut found = 0;
for entry in std::fs::read_dir(dir.path()).expect("lists") {
let text = std::fs::read_to_string(entry.expect("entry").path()).expect("text");
assert!(text.contains("atlas"), "control: {text}");
assert!(text.contains("atlas:"), "control: {text}");
for secret in ["atlas-secret", "argus-secret", "check-secret"] {
assert!(!text.contains(secret), "{secret} reached the disk");
}
found += 1;
}
assert_eq!(found, 2, "only the users file and the list are left behind");
assert_eq!(found, 1, "only the users file is left behind");
}
#[tokio::test]
@ -536,10 +390,7 @@ mod tests {
std::fs::write(&file, text).expect("write");
sync(&held, &["atlas"], &file).await.expect("renders");
let users: Vec<String> = lines(&file).into_iter().map(|(n, _)| n).collect();
assert_eq!(
users,
["atlas", "selfcheck-unrestricted.squid", "selfcheck.squid"]
);
assert_eq!(users, ["atlas", "selfcheck.squid"]);
}
#[tokio::test]
@ -560,17 +411,10 @@ mod tests {
.await
.expect("renders");
let before = std::fs::read_to_string(&file).expect("read");
let list_before = std::fs::read_to_string(list_of(&file)).expect("read");
let list = list_of(&file);
let files = Files {
users: &file,
unrestricted: &list,
};
let failed = sync_with(
HIVE,
&names(&["atlas", "argus"]),
&permissions::Stored::from([flagged("atlas", Some(false))]),
&files,
&file,
|path| async move {
if path.contains("argus") {
bail!("the store is down")
@ -581,11 +425,6 @@ mod tests {
.await;
assert!(failed.is_err());
assert_eq!(std::fs::read_to_string(&file).expect("read"), before);
assert_eq!(
std::fs::read_to_string(&list).expect("read"),
list_before,
"the flag flip waits for every read"
);
}
#[test]
@ -594,124 +433,8 @@ mod tests {
let user = User {
name: bad.to_owned(),
secret: "s".to_owned(),
unrestricted: false,
};
assert!(render("", &[user]).is_err(), "{bad:?}");
}
}
#[tokio::test]
async fn an_agent_is_listed_unless_its_flag_is_off() {
let dir = tempfile::tempdir().expect("tempdir");
let file = dir.path().join("users");
let held = store(&[
("swarm/agents/old/proxy-credential", "s1"),
("swarm/agents/unset/proxy-credential", "s2"),
("swarm/agents/on/proxy-credential", "s3"),
("swarm/agents/new/proxy-credential", "s4"),
("swarm/hives/h1/squid-selfcheck", "check-secret"),
]);
let perms = permissions::Stored::from([
flagged("unset", None),
flagged("on", Some(true)),
flagged("new", Some(false)),
]);
sync_flags(&held, &["old", "unset", "on", "new"], &perms, &file)
.await
.expect("renders");
// `old` has no record at all, `unset` a record without the flag.
assert_eq!(
list(&file),
["old", "on", "selfcheck-unrestricted.squid", "unset"]
);
// Control: the restricted agent still has a login.
let users: Vec<String> = lines(&file).into_iter().map(|(n, _)| n).collect();
assert!(users.contains(&"new".to_owned()), "{users:?}");
}
#[tokio::test]
async fn the_selfchecks_log_in_with_one_secret_and_only_one_is_unrestricted() {
let dir = tempfile::tempdir().expect("tempdir");
let file = dir.path().join("users");
sync(&two_agents_and_a_stranger(), &["atlas"], &file)
.await
.expect("renders");
let by_name: HashMap<_, _> = lines(&file).into_iter().collect();
assert!(verifies("check-secret", &by_name["selfcheck.squid"]));
assert!(verifies(
"check-secret",
&by_name["selfcheck-unrestricted.squid"]
));
let listed = list(&file);
assert!(listed.contains(&"selfcheck-unrestricted.squid".to_owned()));
assert!(
!listed.contains(&"selfcheck.squid".to_owned()),
"{listed:?}"
);
}
#[tokio::test]
async fn an_agent_with_no_credential_is_not_listed() {
let dir = tempfile::tempdir().expect("tempdir");
let file = dir.path().join("users");
sync(&two_agents_and_a_stranger(), &["atlas", "fresh"], &file)
.await
.expect("renders");
let listed = list(&file);
assert!(listed.contains(&"atlas".to_owned()), "control: {listed:?}");
assert!(!listed.contains(&"fresh".to_owned()), "{listed:?}");
}
#[tokio::test]
async fn a_flipped_flag_rewrites_the_list_and_leaves_the_users_file() {
let dir = tempfile::tempdir().expect("tempdir");
let file = dir.path().join("users");
let held = two_agents_and_a_stranger();
let ino =
|p: &Path| std::os::unix::fs::MetadataExt::ino(&std::fs::metadata(p).expect("stat"));
sync(&held, &["atlas", "argus"], &file)
.await
.expect("renders");
assert!(list(&file).contains(&"argus".to_owned()));
let (users_ino, list_ino) = (ino(&file), ino(&list_of(&file)));
// An unchanged pass touches neither.
sync(&held, &["atlas", "argus"], &file)
.await
.expect("renders");
assert_eq!((ino(&file), ino(&list_of(&file))), (users_ino, list_ino));
// Turning argus off changes only the list.
let perms = permissions::Stored::from([flagged("argus", Some(false))]);
sync_flags(&held, &["atlas", "argus"], &perms, &file)
.await
.expect("renders");
assert!(!list(&file).contains(&"argus".to_owned()));
assert!(list(&file).contains(&"atlas".to_owned()));
assert_eq!(ino(&file), users_ino, "the users file was rewritten");
assert_ne!(ino(&list_of(&file)), list_ino, "the list was not");
}
#[tokio::test]
async fn the_list_is_group_readable() {
let dir = tempfile::tempdir().expect("tempdir");
let file = dir.path().join("users");
sync(&two_agents_and_a_stranger(), &["atlas"], &file)
.await
.expect("renders");
let meta = std::fs::metadata(list_of(&file)).expect("stat");
assert_eq!(meta.permissions().mode() & 0o777, 0o640);
}
/// A pass that cannot write the users file has still written the list: a
/// user turned restricted loses its access before the users file stops
/// listing it.
#[tokio::test]
async fn the_list_is_written_before_the_users_file() {
let dir = tempfile::tempdir().expect("tempdir");
let file = dir.path().join("users");
std::fs::create_dir(&file).expect("a directory where the users file belongs");
sync(&two_agents_and_a_stranger(), &["atlas"], &file)
.await
.expect_err("the users file cannot be read");
assert_eq!(list(&file), ["atlas", "selfcheck-unrestricted.squid"]);
}
}

View file

@ -272,11 +272,8 @@ in
BAO_CLIENT_CERT = "%d/bao-client.pem";
BAO_CLIENT_KEY = "%d/bao-client-key.pem";
# Where `workers::squid_users` renders the hashes of this hive's agents' proxy
# credentials for squid, and the list of the users whose agent holds the
# swarm's `unrestricted_internet` flag; it reads the credentials from the
# store under the identity above.
# credentials for squid; it reads them from the store under the identity above.
HIVE_SQUID_USERS_FILE = config.services.hyperhive.squid.usersFile;
HIVE_SQUID_UNRESTRICTED_FILE = config.services.hyperhive.squid.unrestrictedFile;
}
// lib.optionalAttrs (haveBaoClientIdentity && baoDeploy.serverCaFile != null) {
# Absent means the system trust store — right for a deployment with a real

View file

@ -1,12 +1,9 @@
# Egress proxy for this hive's agents: squid on the host, bound to the bridge
# IP, default-deny on the destination hostname. It filters CONNECT targets and
# plain-HTTP hosts and never decrypts, so it holds no signing key. A request
# must carry a per-agent proxy credential, checked against the users file
# hive-c0re renders. A credential's agent either has the swarm's
# `unrestricted_internet` flag, and may then reach any destination the deny
# rules leave, or reaches only the listed names. No agent sends a credential
# yet; the exporter, journal forwarding and the self-check below are how a
# deploy shows it works.
# for a listed name must carry a per-agent proxy credential, checked against the
# users file hive-c0re renders. No agent sends one yet; the exporter, journal
# forwarding and the self-check below are how a deploy shows it works.
{
pkgs,
lib,
@ -27,45 +24,18 @@ let
# The squid user the self-check logs in as: `swarm_secret_client::proxy::SELFCHECK_USER`.
selfcheckUser = "selfcheck.squid";
# The same string in `swarm_secret_client::proxy::SELFCHECK_UNRESTRICTED_USER`.
selfcheckUnrestrictedUser = "selfcheck-unrestricted.squid";
usersDir = builtins.dirOf cfg.usersFile;
# Destinations that are not the public internet: loopback, link-local, the
# bridge, and the private, shared (CGNAT), unspecified, multicast and
# reserved ranges. Squid checks the address it connects to, so a name an
# agent controls cannot point past them.
# `::` is listed because a connect to it lands on loopback; squid's
# built-in `to_localhost` has `0.0.0.0/32` and `::1` but not `::`. NAT64
# (`64:ff9b::/96`) and 6to4 (`2002::/16`) embed IPv4 addresses and are left
# out: they matter only on a host with such a route.
privateRanges = [
"10.0.0.0/8"
"172.16.0.0/12"
"192.168.0.0/16"
"100.64.0.0/10"
"0.0.0.0/8"
"224.0.0.0/4"
"240.0.0.0/4"
"fc00::/7"
"ff00::/8"
"::/128"
];
# Evaluated top to bottom, first match wins. The destination denies sit
# before every allow, `allow unrestricted` included: squid resolves names on
# the host and connects from it, so a name that answers 127.0.0.1,
# 169.254.169.254, the host's own LAN address or a private neighbour would
# otherwise reach services the bridge firewall keeps agents out of, and an
# unrestricted agent is still an agent. The same denies bind a listed name,
# so a listed name must resolve to a public address on the host's resolver.
# The IP-literal
# denies come first because `dstdomain` on an IP literal does a reverse
# lookup. The credential check sits right before the first allow, so only a
# request that would otherwise be let through gets the 407 challenge. An
# unrestricted user is then let through to any other destination on a safe
# port, and every other user to the listed names. The `dst` denies cost an
# unlisted name a lookup on the host's resolver before it is refused.
# Evaluated top to bottom, first match wins. The three destination denies
# sit before the only allow: squid resolves names on the host, so a name
# that answers 127.0.0.1, 169.254.169.254 or a bridge address would
# otherwise reach services the bridge firewall keeps agents out of. The
# unlisted-name deny comes first because `dstdomain` needs no lookup while
# the `dst` ACLs do: an unlisted name is refused at once instead of waiting
# on the host resolver, and costs a client no DNS query. The credential
# check sits right before the allow, so a refused destination answers 403
# to a client with or without a credential, and only a request that would
# otherwise be let through gets the 407 challenge.
squidConf = ''
http_port ${netCfg.bridgeIp}:${toString port}
pid_filename /run/squid/squid.pid
@ -89,17 +59,12 @@ let
acl SSL_ports port 443
acl Safe_ports port 80 443
acl authed proxy_auth REQUIRED
# The users whose agent holds the flag. Squid reads the file when it parses
# its configuration, so `squid-unrestricted-reload` reloads squid when
# hive-c0re rewrites it. An empty or missing file lists no one.
acl unrestricted proxy_auth "${cfg.unrestrictedFile}"
# Host-originated requests: the self-check and the exporter. A local
# connect to the bridge IP takes the bridge IP as its source.
acl selfcheck src ${netCfg.bridgeIp}/32
acl to_bridge dst ${netCfg.networkCidr}
acl to_private dst ${lib.concatStringsSep " " privateRanges}
# `dstdomain` on an IP-literal host does a reverse lookup, so a PTR record
# an agent controls could match an allowed name.
acl ip_v4_host url_regex ^([a-z]+://)?[0-9.]+(:[0-9]+)?(/|$)
@ -117,13 +82,11 @@ let
http_access deny CONNECT !SSL_ports
http_access deny ip_v4_host
http_access deny ip_v6_host
${lib.optionalString (allowed != [ ]) "http_access deny !allowed_dst"}
http_access deny to_localhost
http_access deny to_linklocal
http_access deny to_bridge
http_access deny to_private
http_access deny !authed
http_access allow unrestricted
${lib.optionalString (allowed != [ ]) "http_access deny !allowed_dst"}
${lib.optionalString (allowed != [ ]) "http_access allow allowed_dst"}
http_access deny all
'';
@ -165,32 +128,30 @@ let
secret=$(< "$credential_file")
fi
# <host> <user|anon>: the status squid answers a CONNECT with; 000 when
# the proxy is down. Both users log in with the one secret.
# <host> <auth|anon>: the status squid answers a CONNECT with; 000 when
# the proxy is down.
connect_status() {
{
if [ "$2" != anon ] && [ -n "$secret" ]; then
printf 'proxy-user = "%s:%s"\n' "$2" "$secret"
if [ "$2" = auth ] && [ -n "$secret" ]; then
printf 'proxy-user = "%s:%s"\n' ${selfcheckUser} "$secret"
fi
} | curl -sS -m 10 -K - --interface ${netCfg.bridgeIp} -p -x ${proxyUrl} \
-o /dev/null -w '%{http_connect}' "https://$1/" 2>/dev/null || true
}
allow_status=$(connect_status ${cfg.selfcheck.canary} ${selfcheckUser})
deny_status=$(connect_status ${denyCanary} ${selfcheckUser})
allow_status=$(connect_status ${cfg.selfcheck.canary} auth)
deny_status=$(connect_status ${denyCanary} auth)
anon_status=$(connect_status ${cfg.selfcheck.canary} anon)
unrestricted_status=$(connect_status ${cfg.selfcheck.unrestrictedCanary} ${selfcheckUnrestrictedUser})
ok=false
if [ "$allow_status" = 200 ] && [ "$deny_status" = 403 ] && [ "$anon_status" = 407 ] \
&& [ "$unrestricted_status" = 200 ]; then
if [ "$allow_status" = 200 ] && [ "$deny_status" = 403 ] && [ "$anon_status" = 407 ]; then
ok=true
fi
credential=absent
if [ -n "$secret" ]; then
credential=present
fi
echo "squid-selfcheck ok=$ok allow_canary=${cfg.selfcheck.canary} allow_status=$allow_status deny_canary=${denyCanary} deny_status=$deny_status anon_status=$anon_status credential=$credential unrestricted_canary=${cfg.selfcheck.unrestrictedCanary} unrestricted_status=$unrestricted_status"
echo "squid-selfcheck ok=$ok allow_canary=${cfg.selfcheck.canary} allow_status=$allow_status deny_canary=${denyCanary} deny_status=$deny_status anon_status=$anon_status credential=$credential"
[ "$ok" = true ]
'';
};
@ -207,12 +168,8 @@ in
];
description = ''
Hostnames squid lets a client reach, by CONNECT or plain HTTP. Each
entry matches that exact name. Everything else gets a 403, except
for an unrestricted user. The self-check canary is allowed on top of
this list. A name must resolve, on this host's resolver, to a public
address: one in a loopback, link-local, private, shared (100.64.0.0/10),
multicast or reserved range (or is `::`), or in the bridge subnet, gets a 403 for
every user.
entry matches that exact name. Everything else gets a 403. The
self-check canary is allowed on top of this list.
'';
};
@ -229,23 +186,6 @@ in
'';
};
unrestrictedFile = lib.mkOption {
type = lib.types.path;
default = "${builtins.dirOf cfg.usersFile}/unrestricted";
readOnly = true;
defaultText = lib.literalMD "`unrestricted`, in the directory of `usersFile`";
description = ''
The user names, one per line, that squid lets reach any destination the
deny rules leave open: the users of the agents that hold the swarm's
`unrestricted_internet` flag (set unless the swarm records it off), and
the self-check's unrestricted user. hive-c0re renders it beside
`usersFile`, and squid is reloaded when it changes. Every other user
reaches only `allowedDomains`. An empty file, which is what exists
until hive-c0re writes one, makes every user a restricted one.
Owned by `hive-core` and readable by the `squid` group.
'';
};
selfcheck.credentialName = lib.mkOption {
type = lib.types.str;
default = "hive-squid.selfcheck";
@ -258,17 +198,6 @@ in
'';
};
selfcheck.unrestrictedCanary = lib.mkOption {
type = lib.types.str;
default = "example.com";
description = ''
Hostname outside `allowedDomains` that the self-check CONNECTs to on
port 443, as its unrestricted user, and expects squid to let through.
Squid resolves it with this host's resolver and connects to it, so a
host with no route to it reports `ok=false`.
'';
};
selfcheck.canary = lib.mkOption {
type = lib.types.str;
default = hyperhiveCfg.swarm.forge.domain;
@ -276,10 +205,9 @@ in
description = ''
Hostname the self-check CONNECTs to on port 443 and expects squid to
let through. Squid resolves it with this host's resolver, not the
hive's: a host whose resolver answers a loopback, bridge or other
private address for it (a single-host swarm with the gateway's
`localHostsEntry`, or a split-horizon LAN) gets a 403, and the
self-check reports `ok=false`.
hive's: a host whose resolver answers a loopback or bridge address
for it (a single-host swarm with the gateway's `localHostsEntry`) gets
a 403, and the self-check reports `ok=false`.
'';
};
};
@ -302,43 +230,11 @@ in
};
# The directory is setgid so the file hive-c0re renames into it is the
# squid group's. The users file exists before squid starts, empty: the
# helper exits on a missing file and squid then rejects every login. The
# unrestricted list is created the same way, though squid treats a missing
# one as empty.
# squid group's. The file exists before squid starts, empty: the helper
# exits on a missing file and squid then rejects every login.
systemd.tmpfiles.rules = [
"d ${usersDir} 2750 hive-core squid - -"
"f ${cfg.usersFile} 0640 hive-core squid - -"
"f ${cfg.unrestrictedFile} 0640 hive-core squid - -"
];
# Squid reads the unrestricted list when it parses its configuration, not
# when it changes. hive-c0re writes the list only when its contents change,
# so each change is one reload. Nothing happens to a squid that is not
# running; it reads the list when it starts.
systemd.paths.squid-unrestricted-reload = {
description = "Watch squid's unrestricted-users list";
wantedBy = [ "paths.target" ];
pathConfig.PathChanged = cfg.unrestrictedFile;
};
systemd.services.squid-unrestricted-reload = {
description = "Reload squid to read its unrestricted-users list";
serviceConfig = {
Type = "oneshot";
ExecStart = "${config.systemd.package}/bin/systemctl try-reload-or-restart squid.service";
};
};
assertions = [
{
assertion = !(builtins.elem cfg.selfcheck.unrestrictedCanary allowed);
message = ''
services.hyperhive.squid.selfcheck.unrestrictedCanary must be a name outside
allowedDomains, or the self-check cannot tell an unrestricted user from a
restricted one.
'';
}
];
systemd.services.squid = {

View file

@ -485,110 +485,38 @@ let
&& !(lib.hasInfix "localnet" conf);
}
{
# First match wins, so a destination deny after an allow is dead, and a
# credential check after one never challenges anyone. `allow unrestricted`
# is an allow: above the three destination denies an unrestricted user
# reaches loopback, link-local and the bridge through the proxy. The two
# IP-literal denies stay ahead of every allow: `dstdomain` on an IP
# First match wins, so a destination deny after the allow is dead, and
# a credential check after it never challenges anyone. The two IP-literal
# denies stay ahead of the unlisted-name deny: `dstdomain` on an IP
# literal does a reverse lookup.
name = "squid refuses loopback, link-local, bridge and private destinations and IP literals, then a missing credential, before any allow";
name = "squid refuses unlisted names, then loopback, link-local and bridge destinations, then a missing credential, before any allow";
ok =
let
lines = lib.splitString "\n" hiveOtel.services.squid.configText;
at = l: lib.lists.findFirstIndex (x: x == l) null lines;
allow = at "http_access allow allowed_dst";
unrestricted = at "http_access allow unrestricted";
authed = at "http_access deny !authed";
firstAllow = if unrestricted < allow then unrestricted else allow;
in
allow != null
&& unrestricted != null
&& authed != null
&& at "http_access deny to_localhost" != null
&& at "http_access deny to_localhost" < allow
&& at "http_access deny to_linklocal" != null
&& at "http_access deny to_linklocal" < allow
&& at "http_access deny to_bridge" != null
&& at "http_access deny to_private" != null
&& at "http_access deny ip_v4_host" != null
&& at "http_access deny ip_v6_host" != null
&& lib.all (l: at l < firstAllow) [
"http_access deny to_localhost"
"http_access deny to_linklocal"
"http_access deny to_bridge"
"http_access deny to_private"
"http_access deny ip_v4_host"
"http_access deny ip_v6_host"
]
&& at "http_access deny to_bridge" < allow
&& at "http_access deny to_localhost" < authed
&& at "http_access deny to_linklocal" < authed
&& at "http_access deny to_bridge" < authed
&& at "http_access deny to_private" < authed
&& authed < firstAllow
&& authed < allow
&& at "http_access deny ip_v4_host" != null
&& at "http_access deny ip_v4_host" < at "http_access deny !allowed_dst"
&& at "http_access deny ip_v6_host" < at "http_access deny !allowed_dst"
&& at "http_access deny !allowed_dst" != null
&& at "http_access deny !allowed_dst" < at "http_access deny to_localhost"
&& builtins.elem "cache deny all" lines
&& lib.last (lib.filter (l: l != "") lines) == "http_access deny all";
}
{
# The ranges are what keeps an unrestricted user on the public internet.
# Control: the bridge deny and the ACL line are found, so a range missing
# below is not a lookup of nothing.
name = "squid's private-destination ACL names every private, shared, multicast and reserved range";
ok =
let
lines = lib.splitString "\n" hiveOtel.services.squid.configText;
acl = lib.findFirst (l: lib.hasPrefix "acl to_private dst " l) null lines;
ranges = lib.splitString " " (lib.removePrefix "acl to_private dst " acl);
in
acl != null
&& builtins.elem "acl to_bridge dst ${hiveOtel.services.hyperhive.network.networkCidr}" lines
&&
lib.sort lib.lessThan ranges == lib.sort lib.lessThan [
"10.0.0.0/8"
"172.16.0.0/12"
"192.168.0.0/16"
"100.64.0.0/10"
"0.0.0.0/8"
"224.0.0.0/4"
"240.0.0.0/4"
"fc00::/7"
"ff00::/8"
"::/128"
];
}
{
# An unrestricted user is let through before the unlisted-name deny and
# everyone else is held to the listed names after it. Control: both are
# there, so the ordering below is not a lookup of nothing.
name = "squid lets an unrestricted user through before the allow-list rules that bind everyone else";
ok =
let
conf = hiveOtel.services.squid.configText;
lines = lib.splitString "\n" conf;
at = l: lib.lists.findFirstIndex (x: x == l) null lines;
list = hiveOtel.services.hyperhive.squid.unrestrictedFile;
in
at "http_access allow unrestricted" != null
&& at "http_access deny !allowed_dst" != null
&& at "http_access allow unrestricted" < at "http_access deny !allowed_dst"
&& at "http_access deny !allowed_dst" < at "http_access allow allowed_dst"
&& builtins.elem ''acl unrestricted proxy_auth "${list}"'' lines
&& list == "/var/lib/hive-squid/unrestricted";
}
{
# The list is read when squid parses its configuration, so a change to
# the file reloads squid. Control: the watched path is the list's own.
name = "squid is reloaded when its unrestricted-users list changes";
ok =
let
list = hiveOtel.services.hyperhive.squid.unrestrictedFile;
rules = hiveOtel.systemd.tmpfiles.rules;
path = hiveOtel.systemd.paths.squid-unrestricted-reload;
svc = hiveOtel.systemd.services.squid-unrestricted-reload.serviceConfig;
in
path.pathConfig.PathChanged == list
&& builtins.elem "paths.target" path.wantedBy
&& lib.hasSuffix "systemctl try-reload-or-restart squid.service" svc.ExecStart
&& builtins.elem "f ${list} 0640 hive-core squid - -" rules
&& hiveOtelAllLocal.systemd.services.hive-c0re.environment.HIVE_SQUID_UNRESTRICTED_FILE == list;
}
{
# The helper is the NCSA one, on the file hive-c0re renders. The file
# exists, empty, before squid starts (the helper exits on a missing one),

View file

@ -66,38 +66,6 @@ let
config.services.hyperhive.network.exposeHostPorts;
};
# The uplink is a public range. The private ranges squid must refuse are
# extra subnets on the same link, so the proxy can reach a listener in each:
# a name that resolves there is refused only by squid's own rules.
privateNets = [
{
name = "priv10.t.local";
net = "10.99.0";
}
{
name = "priv172.t.local";
net = "172.16.99";
}
{
name = "priv192.t.local";
net = "192.168.99";
}
{
name = "cgnat.t.local";
net = "100.64.99";
}
];
privateAddrs =
host:
map (p: {
address = "${p.net}.${toString host}";
prefixLength = 24;
}) privateNets;
ulaAddr = host: {
address = "fd00:99::${toString host}";
prefixLength = 64;
};
# Plain TCP on 443: a CONNECT is answered before any TLS, which is all the
# test reads.
listen443 = addrs: {
@ -120,10 +88,9 @@ pkgs.testers.runNixOSTest {
imports = [
hiveStubs
../host-modules/hive-squid.nix
# Listeners a leaked destination would reach: loopback (`::` lands on `::1`) and the bridge.
# Listeners a leaked destination would reach: loopback and the bridge.
(listen443 [
"127.0.0.1"
"[::1]"
bridgeIp
])
];
@ -137,73 +104,40 @@ pkgs.testers.runNixOSTest {
prefixLength = 24;
}
];
networking.interfaces.eth2.ipv4.addresses = lib.mkForce (
[
{
address = "203.0.113.1";
prefixLength = 24;
}
]
++ privateAddrs 1
);
networking.interfaces.eth2.ipv6.addresses = [ (ulaAddr 1) ];
networking.interfaces.eth2.ipv4.addresses = lib.mkForce [
{
address = "192.168.2.1";
prefixLength = 24;
}
];
# Names the proxy resolves, each a different destination class.
networking.hosts = {
# `other.t.local` and the self-check's deny canary resolve, are
# reachable and are on no allow-list. The canary is in the hosts file
# because the destination denies look every name up first, and the VM
# has no resolver to answer for an unknown one before the client gives up.
"203.0.113.2" = [
"forge.t.local"
"other.t.local"
"deny-canary.invalid"
];
"192.168.2.2" = [ "forge.t.local" ];
"127.0.0.1" = [ "loop.t.local" ];
${bridgeIp} = [ "bridge.t.local" ];
"169.254.169.254" = [ "meta.t.local" ];
"fd00:99::2" = [ "ula.t.local" ];
"::" = [ "any6.t.local" ];
}
// lib.listToAttrs (
map (p: {
name = "${p.net}.2";
value = [ p.name ];
}) privateNets
);
};
# Renders the users file the way hive-c0re does.
environment.systemPackages = [ pkgs.mkpasswd ];
services.hyperhive.squid.selfcheck.unrestrictedCanary = "other.t.local";
services.hyperhive.squid.allowedDomains = [
"forge.t.local"
"loop.t.local"
"bridge.t.local"
"meta.t.local"
"ula.t.local"
"any6.t.local"
]
++ map (p: p.name) privateNets;
];
};
web =
{ lib, ... }:
{
imports = [
(listen443 [
"0.0.0.0"
"[::]"
])
];
imports = [ (listen443 [ "0.0.0.0" ]) ];
virtualisation.vlans = [ 2 ];
networking.interfaces.eth1.ipv4.addresses = lib.mkForce (
[
{
address = "203.0.113.2";
prefixLength = 24;
}
]
++ privateAddrs 2
);
networking.interfaces.eth1.ipv6.addresses = [ (ulaAddr 2) ];
networking.interfaces.eth1.ipv4.addresses = lib.mkForce [
{
address = "192.168.2.2";
prefixLength = 24;
}
];
networking.firewall.allowedTCPPorts = [ 443 ];
};
@ -223,24 +157,14 @@ pkgs.testers.runNixOSTest {
testScript = ''
import re
import secrets
import time
# Drawn fresh each run. The users file holds only their hashes, as
# hive-c0re renders it. `atlas` is a restricted agent and `uma` an
# unrestricted one; the two self-check users share one secret.
PW = {
u: secrets.token_urlsafe(24)
for u in ("atlas", "uma", "selfcheck.squid", "ruth")
}
PW["selfcheck-unrestricted.squid"] = PW["selfcheck.squid"]
# hive-c0re renders it.
PW = {u: secrets.token_urlsafe(24) for u in ("atlas", "selfcheck.squid", "ruth")}
ATLAS = "atlas:" + PW["atlas"]
UMA = "uma:" + PW["uma"]
SELFCHECK = "selfcheck.squid:" + PW["selfcheck.squid"]
SELFCHECK_U = "selfcheck-unrestricted.squid:" + PW["selfcheck.squid"]
RUTH = "ruth:" + PW["ruth"]
USERS = "/var/lib/hive-squid/users"
LIST = "/var/lib/hive-squid/unrestricted"
UNRESTRICTED = ["uma", "selfcheck-unrestricted.squid"]
start_all()
proxy.wait_for_unit("squid.service")
@ -249,8 +173,6 @@ pkgs.testers.runNixOSTest {
web.wait_for_unit("nginx.service")
proxy.wait_for_open_port(3128, "${bridgeIp}")
PRIVATE = ["priv10.t.local", "priv172.t.local", "priv192.t.local", "cgnat.t.local", "ula.t.local", "any6.t.local"]
def connect(host, cred=None):
# The status squid answered the CONNECT with; 000 when none came back.
auth = f"-U '{cred}' " if cred else ""
@ -272,47 +194,17 @@ pkgs.testers.runNixOSTest {
f"&& chmod 640 {USERS}.new && mv {USERS}.new {USERS} && sleep 1"
)
def write_list(*names):
# What hive-c0re does for the list. Squid reads it on a reload, which
# squid-unrestricted-reload does when the file changes.
body = "printf '%s\\n' " + " ".join(names) if names else ":"
proxy.succeed(
f"({body}) > {LIST}.new && chown hive-core:squid {LIST}.new "
f"&& chmod 640 {LIST}.new && mv {LIST}.new {LIST}"
)
def wait_status(host, cred, want):
# The reload that follows a list write is asynchronous.
got = None
for _ in range(30):
got = connect(host, cred)
if got == want:
return
time.sleep(1)
raise AssertionError(f"{host} as {cred.split(':')[0]}: wanted {want}, last got {got}")
with subtest("the private listeners answer the proxy itself, so a refusal below is squid's"):
for addr in ["10.99.0.2", "172.16.99.2", "192.168.99.2", "100.64.99.2", "fd00:99::2"]:
proxy.wait_until_succeeds(f"timeout 3 bash -c 'echo > /dev/tcp/{addr}/443'")
assert connect("forge.t.local", None) == "407", "control: the public name is challenged, not refused"
with subtest("squid starts on the empty users file and list and refuses every login"):
with subtest("squid starts on the empty users file and refuses every login"):
assert proxy.succeed(f"stat -c '%U:%G %a' {USERS}").strip() == "hive-core:squid 640"
assert proxy.succeed(f"stat -c '%U:%G %a' {LIST}").strip() == "hive-core:squid 640"
assert proxy.succeed("stat -c %a /var/lib/hive-squid").strip() == "2750"
assert proxy.succeed(f"stat -c %s {USERS}").strip() == "0"
assert proxy.succeed(f"stat -c %s {LIST}").strip() == "0"
assert connect("forge.t.local", ATLAS) == "407", "no such user"
assert connect("other.t.local", UMA) == "407", "no such user, unlisted name"
assert connect("deny-canary.invalid", ATLAS) == "403", "an unlisted name needs no login"
with subtest("the users file holds hashes, never a password"):
write_users(ATLAS, UMA, SELFCHECK, SELFCHECK_U)
write_users(ATLAS, SELFCHECK)
users = proxy.succeed(f"cat {USERS}")
rows = [l.split(":", 1) for l in users.splitlines()]
assert sorted(r[0] for r in rows) == sorted(
["atlas", "uma", "selfcheck.squid", "selfcheck-unrestricted.squid"]
), users
assert all(re.fullmatch(r"\$6\$\S+", r[1]) for r in rows), users
assert re.fullmatch(r"(atlas|selfcheck\.squid):\$6\$\S+\n(atlas|selfcheck\.squid):\$6\$\S+\n", users), users
assert not any(p in users for p in PW.values()), users
with subtest("an allowed name is tunnelled with a credential and challenged without one"):
@ -323,33 +215,22 @@ pkgs.testers.runNixOSTest {
# DES would accept the first eight characters; the hash must not.
assert connect("forge.t.local", "atlas:" + PW["atlas"][:8]) == "407", "a prefix of the password"
with subtest("with an empty list every user is restricted"):
assert connect("other.t.local", UMA) == "403", "uma, not listed yet"
assert connect("other.t.local", ATLAS) == "403"
assert connect("forge.t.local", UMA) == "200", "control: uma still reaches a listed name"
with subtest("an unlisted name is refused with or without a credential"):
assert connect("deny-canary.invalid", ATLAS) == "403", "deny canary"
assert connect("deny-canary.invalid") == "403", "deny canary, anonymous"
with subtest("a listed user reaches an unlisted name and the others are refused"):
write_list(*UNRESTRICTED)
wait_status("other.t.local", UMA, "200")
assert connect("other.t.local", ATLAS) == "403", "restricted"
assert connect("forge.t.local", ATLAS) == "200", "restricted, listed name"
assert connect("forge.t.local", UMA) == "200", "unrestricted, listed name"
assert connect("other.t.local") == "407", "no credential"
assert connect("other.t.local", "uma:" + secrets.token_urlsafe(24)) == "407", "wrong password"
assert connect("deny-canary.invalid", ATLAS) == "403", "restricted, an unlisted name"
with subtest("a name resolving to a host-local destination is refused"):
assert connect("loop.t.local", ATLAS) == "403", "loopback"
assert connect("bridge.t.local", ATLAS) == "403", "bridge address"
assert connect("meta.t.local", ATLAS) == "403", "link-local"
with subtest("neither kind of user reaches a host-local or private destination, or another port"):
for cred in (ATLAS, UMA):
who = cred.split(":")[0]
assert connect("loop.t.local", cred) == "403", f"{who}: loopback"
assert connect("bridge.t.local", cred) == "403", f"{who}: bridge address"
assert connect("meta.t.local", cred) == "403", f"{who}: link-local"
assert connect("127.0.0.1", cred) == "403", f"{who}: loopback literal"
assert connect("203.0.113.2", cred) == "403", f"{who}: IP literal"
for name in PRIVATE:
assert connect(name, cred) == "403", f"{who}: {name}, a listed name that resolves to a private address"
assert connect("other.t.local:22", cred) == "403", f"{who}: port 22"
assert connect("other.t.local:80", cred) == "403", f"{who}: CONNECT to port 80"
with subtest("IP literals are refused even when the address is reachable"):
assert connect("127.0.0.1", ATLAS) == "403"
assert connect("192.168.2.2", ATLAS) == "403"
with subtest("only a port-443 CONNECT is allowed"):
assert connect("forge.t.local:22", ATLAS) == "403"
assert connect("forge.t.local:80", ATLAS) == "403"
with subtest("an agent cannot read cachemgr"):
agent.succeed(
@ -366,50 +247,32 @@ pkgs.testers.runNixOSTest {
"journalctl -u squid-selfcheck.service -o cat | "
"grep -F 'squid-selfcheck ok=true allow_canary=forge.t.local "
"allow_status=200 deny_canary=deny-canary.invalid deny_status=403 "
"anon_status=407 credential=present "
"unrestricted_canary=other.t.local unrestricted_status=200'"
"anon_status=407 credential=present'"
)
with subtest("the self-check fails, and says why, when the unrestricted user is not listed"):
write_list()
wait_status("other.t.local", UMA, "403")
proxy.fail("systemctl start squid-selfcheck.service")
proxy.succeed(
"journalctl -u squid-selfcheck.service -o cat | "
"grep -F 'squid-selfcheck ok=false allow_canary=forge.t.local "
"allow_status=200 deny_canary=deny-canary.invalid deny_status=403 "
"anon_status=407 credential=present "
"unrestricted_canary=other.t.local unrestricted_status=403'"
)
write_list(*UNRESTRICTED)
wait_status("other.t.local", UMA, "200")
with subtest("the self-check fails, and says why, without a credential or with the wrong one"):
proxy.succeed("mv /run/credstore/hive-squid.selfcheck /run/credstore/kept")
proxy.fail("systemctl start squid-selfcheck.service")
proxy.succeed(
"journalctl -u squid-selfcheck.service -o cat | "
"grep -F 'squid-selfcheck ok=false allow_canary=forge.t.local "
"allow_status=407 deny_canary=deny-canary.invalid deny_status=407 "
"anon_status=407 credential=absent "
"unrestricted_canary=other.t.local unrestricted_status=407'"
"allow_status=407 deny_canary=deny-canary.invalid deny_status=403 "
"anon_status=407 credential=absent'"
)
proxy.succeed(f"printf '%s\\n' {secrets.token_urlsafe(24)} > /run/credstore/hive-squid.selfcheck")
proxy.fail("systemctl start squid-selfcheck.service")
proxy.succeed(
"journalctl -u squid-selfcheck.service -o cat | "
"grep -F 'allow_status=407 deny_canary=deny-canary.invalid deny_status=407 "
"anon_status=407 credential=present "
"unrestricted_canary=other.t.local unrestricted_status=407'"
"grep -F 'allow_status=407 deny_canary=deny-canary.invalid deny_status=403 "
"anon_status=407 credential=present'"
)
proxy.succeed("mv -f /run/credstore/kept /run/credstore/hive-squid.selfcheck")
with subtest("squid's access log reaches the journal under squid.service, with the user"):
journal = proxy.succeed("journalctl -u squid.service -o cat")
assert re.search(r"192\.168\.1\.3 TCP_TUNNEL/200 .* CONNECT forge\.t\.local:443 atlas ", journal), journal
assert re.search(r"192\.168\.1\.3 TCP_TUNNEL/200 .* CONNECT other\.t\.local:443 uma ", journal), journal
assert re.search(r"192\.168\.1\.3 TCP_DENIED/407 .* CONNECT forge\.t\.local:443 - ", journal), journal
assert re.search(r"192\.168\.1\.3 TCP_DENIED/403 .* CONNECT other\.t\.local:443 atlas ", journal), journal
assert re.search(r"192\.168\.1\.3 TCP_DENIED/403 .* CONNECT deny-canary\.invalid:443 ", journal), journal
assert re.search(r"192\.168\.1\.3 TCP_DENIED/403 .* CONNECT loop\.t\.local:443 ", journal), journal
# Exporter scrapes are cachemgr requests and stay out of the log.
assert "squid-internal-mgr" not in journal, journal
@ -424,36 +287,15 @@ pkgs.testers.runNixOSTest {
assert m and float(m.group(1)) >= 1, metrics
proxy.fail("curl -sf -m 5 http://${bridgeIp}:9301/metrics")
with subtest("squid survives a reload and keeps the list"):
with subtest("squid survives a reload"):
proxy.succeed("systemctl reload squid.service")
proxy.wait_for_unit("squid.service")
assert connect("forge.t.local", ATLAS) == "200"
assert connect("other.t.local", UMA) == "200"
assert connect("other.t.local", ATLAS) == "403"
with subtest("a user added while squid runs can log in, with no reload, and is restricted"):
write_users(ATLAS, UMA, SELFCHECK, SELFCHECK_U, RUTH)
with subtest("a user added while squid runs can log in, with no reload"):
write_users(ATLAS, SELFCHECK, RUTH)
assert connect("forge.t.local", RUTH) == "200"
assert connect("forge.t.local", ATLAS) == "200"
assert connect("other.t.local", RUTH) == "403"
with subtest("a user the list gains is unrestricted after the reload and loses it again"):
write_list(*UNRESTRICTED, "ruth")
wait_status("other.t.local", RUTH, "200")
assert connect("loop.t.local", RUTH) == "403", "still no loopback"
write_list(*UNRESTRICTED)
wait_status("other.t.local", RUTH, "403")
assert connect("other.t.local", UMA) == "200", "control: uma is still listed"
with subtest("squid starts on a missing list, treats every user as restricted, and recovers when it appears"):
proxy.succeed(f"rm {LIST}")
proxy.succeed("systemctl restart squid.service")
proxy.wait_for_unit("squid.service")
proxy.wait_for_open_port(3128, "${bridgeIp}")
assert connect("forge.t.local", UMA) == "200", "logins work"
assert connect("other.t.local", UMA) == "403", "no list, no unrestricted user"
write_list(*UNRESTRICTED)
wait_status("other.t.local", UMA, "200")
with subtest("squid starts on a missing users file, refuses logins, and recovers when it appears"):
proxy.succeed(f"rm {USERS}")
@ -461,9 +303,8 @@ pkgs.testers.runNixOSTest {
proxy.wait_for_unit("squid.service")
proxy.wait_for_open_port(3128, "${bridgeIp}")
assert connect("forge.t.local", ATLAS) == "407", "no file, no login"
assert connect("other.t.local", UMA) == "407", "no file, no login, even when listed"
write_users(ATLAS, UMA, SELFCHECK, SELFCHECK_U)
assert connect("deny-canary.invalid", ATLAS) == "403", "an unlisted name still answers"
write_users(ATLAS, SELFCHECK)
assert connect("forge.t.local", ATLAS) == "200", "the file appeared"
assert connect("other.t.local", UMA) == "200", "listed, and now logged in"
'';
}

View file

@ -28,14 +28,6 @@ pub const SELFCHECK_LEAF: &str = "squid-selfcheck";
/// `nix/host-modules/hive-squid.nix` spells the same string.
pub const SELFCHECK_USER: &str = "selfcheck.squid";
/// The squid user the self-check logs in as to prove an unrestricted agent
/// reaches an unlisted name. It authenticates with the same secret as
/// [`SELFCHECK_USER`] ([`selfcheck_path`]), and is always on squid's
/// unrestricted list. It cannot be an agent's name for the same reason as
/// [`SELFCHECK_USER`]. `nix/host-modules/hive-squid.nix` spells the same
/// string.
pub const SELFCHECK_UNRESTRICTED_USER: &str = "selfcheck-unrestricted.squid";
/// The path holding the secret `agent` authenticates to the egress proxy with.
///
/// A flat leaf under the agent's prefix, like
@ -135,13 +127,8 @@ mod tests {
}
#[test]
fn the_selfcheck_users_are_not_legal_agent_names() {
for user in [SELFCHECK_USER, SELFCHECK_UNRESTRICTED_USER] {
assert!(
crate::path::checked_segment("agent", user).is_err(),
"{user}"
);
}
fn the_selfcheck_user_is_not_a_legal_agent_name() {
assert!(crate::path::checked_segment("agent", SELFCHECK_USER).is_err());
}
#[test]