squid blocks a runtime deny list from the controller in every egress mode, ahead of the allow list
This commit is contained in:
atlas
mara
parent
99ab0ae77f
commit
dd5fde848f
21 changed files with 1814 additions and 24 deletions
|
|
@ -309,6 +309,60 @@ host. It never decrypts TLS, so it holds no signing key. It caches nothing.
|
|||
time or a count. The self-check's users aren't agents,
|
||||
so `deny-canary.invalid` stays a 403.
|
||||
|
||||
- **Modes.** An agent is in one of three modes, and the deny list (next bullet)
|
||||
applies in all of them:
|
||||
|
||||
| mode | the deny list | a name not on the allow list |
|
||||
| -------------- | ------------- | ---------------------------- |
|
||||
| `restricted` | blocked | blocked |
|
||||
| `log` | blocked | allowed and logged |
|
||||
| `unrestricted` | blocked | allowed |
|
||||
|
||||
- **Deny list.** The swarm's deny list names destinations the proxy refuses
|
||||
whatever else allows them. The swarm controller keeps it as its own value
|
||||
(the `deny` key of the `egress-allow` bucket), each entry for every agent or
|
||||
for one, and serves it at `GET`/`POST`/`DELETE /api/egress/denied-domains`
|
||||
with the same body as the allow-list (`{"domain": ".example.com", "agent":
|
||||
"atlas"}`). Write a name as in the allow-list: at least two labels, one
|
||||
leading dot for the name and its subdomains. The controller refuses IP
|
||||
addresses, other wildcards, ports and paths. Unlike the allow-list it doesn't
|
||||
refuse the base list's names: denying one is the point. Every 30 seconds
|
||||
`hive-c0re` reads the list and renders `services.hyperhive.deploy.hive-controller.squid.denyFile`
|
||||
(`/var/lib/hive-squid/deny.conf`, `hive-core:squid`, `0640`) for the agents it
|
||||
runs, and `squid-unrestricted-reload.path` reloads squid when it changes.
|
||||
- **Precedence.** The configuration includes the file after the destination
|
||||
denies and before `deny !authed` and every allow, so a denied name is
|
||||
refused even when the base list, the hive's `allowedDomains`, the swarm's
|
||||
allow-list, the unrestricted flag or `log` mode would allow it. A swarm-wide
|
||||
entry names no user, so every client gets a 403 for it, one without a
|
||||
credential too. An agent's entry also names the agent's squid user, so it
|
||||
binds that agent: a client that sent no credential gets the usual 407 for
|
||||
it. An agent's entry only adds to the swarm-wide ones; no entry of one scope
|
||||
lifts another.
|
||||
- **Log line.** A request a deny rule refused gets a second log line,
|
||||
`egress_denied user=<agent or -> method=<m> host=<h> port=<p> result=<r>`,
|
||||
in every mode. It differs from the `would_deny` line of `log` mode and
|
||||
from the `TCP_DENIED/403` line every refusal has, so you can tell a
|
||||
deny-list hit from a name that isn't on the allow-list.
|
||||
- **Who writes it.** Only an operator's request to the controller writes the
|
||||
list. It has no automatic entries: nothing derives a deny from an agent's
|
||||
API-key environment or a linked account, and the stored type has no such
|
||||
field. An agent has no queue grant that reaches the key, and the controller's
|
||||
API sits behind the operator's SSO login: nginx checks it before it hands
|
||||
a request to the socket, which only nginx and root can open.
|
||||
- **Empty list.** With no entry `deny.conf` is empty and squid configures as
|
||||
it did without the list: the configuration gains one `include` line and the
|
||||
`egress_denied` log lines, and no rule. A hive on a `hive-c0re` that
|
||||
predates the list never reads the key and keeps its old configuration, so
|
||||
nothing on its wire changes.
|
||||
- **Applied.** Each hive reports the revision of the list it last rendered as
|
||||
`egress_deny_applied` in its status snapshot (see [the hive
|
||||
snapshot](../swarm/README.md)). `GET /api/egress/denied-domains` answers the
|
||||
list's `revision` and `behind`, the hives that haven't reported it: an older
|
||||
`hive-c0re` never does, so it shows there rather than silently allowing the
|
||||
names. A hive that reads the list reports within about a minute. Reporting
|
||||
means the file holds the rules; squid reloads moments later.
|
||||
|
||||
- **Base list.** `services.hyperhive.deploy.hive-controller.squid.baseAllowedDomains` is open to every
|
||||
authenticated user, whatever its flag: what the agent harness needs off the
|
||||
swarm. The default is the hosts Claude Code requires
|
||||
|
|
@ -419,7 +473,7 @@ host. It never decrypts TLS, so it holds no signing key. It caches nothing.
|
|||
`squid.service`, and `services.hyperhive.otel.journalUnits` forwards that
|
||||
unit and the self-check's to the swarm's log store. A refusal shows as `TCP_DENIED/403` with the
|
||||
client's source address. A request that a `log`-mode agent's rule let
|
||||
through also logs a `would_deny` line (see _Log mode_).
|
||||
through also logs a `would_deny` line (see _Log mode_). A request a deny-list rule refused logs an `egress_denied` line (see _Deny list_).
|
||||
- **Self-check.** `squid-selfcheck.timer` runs `squid-selfcheck.service`
|
||||
every minute from the bridge IP, logging in as `selfcheck.squid`: `CONNECT`
|
||||
to `services.hyperhive.deploy.hive-controller.squid.selfcheck.canary` (default: `cache.nixos.org`,
|
||||
|
|
|
|||
|
|
@ -99,6 +99,11 @@ of names for what the hive's release can do. Two exist: `tolerant-permissions`,
|
|||
[turning an archetype agent into a custom one](archetypes.md#turn-into-custom)
|
||||
needs. A hive on an older release lists none.
|
||||
|
||||
The snapshot also carries `egress_deny_applied`, the revision of the egress deny
|
||||
list the hive last rendered into its proxy (`0` when the swarm has none). A hive
|
||||
on an older release, or one that hasn't read the list yet, leaves it out; see
|
||||
[the egress proxy](../networking/network.md#egress-proxy-squid).
|
||||
|
||||
Every row also carries `last_seen_unix` and `age_seconds` if you want to
|
||||
apply your own threshold. The timestamp is the one the queue recorded on
|
||||
arrival, not one the hive put in its own payload. A roster hive's row
|
||||
|
|
|
|||
|
|
@ -122,6 +122,12 @@ pub fn spawn(
|
|||
shutdown.clone(),
|
||||
));
|
||||
|
||||
// The swarm's egress deny list, on the same connection.
|
||||
tokio::spawn(crate::workers::squid_deny::follow(
|
||||
client.clone(),
|
||||
shutdown.clone(),
|
||||
));
|
||||
|
||||
// The permissions the swarm grants, and the import of the meta repo's.
|
||||
tokio::spawn(crate::workers::swarm_permissions::follow(
|
||||
client.clone(),
|
||||
|
|
@ -375,15 +381,24 @@ struct Snapshot {
|
|||
#[serde(flatten)]
|
||||
readiness: crate::warnings::Readiness,
|
||||
capabilities: &'static [&'static str],
|
||||
/// The deny-list revision this hive rendered into squid's include
|
||||
/// (`swarm_wire::hive_status::EGRESS_DENY_APPLIED`); absent until a pass has.
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
egress_deny_applied: Option<u64>,
|
||||
}
|
||||
|
||||
fn snapshot() -> Snapshot {
|
||||
snapshot_with(crate::workers::squid_deny::applied())
|
||||
}
|
||||
|
||||
fn snapshot_with(egress_deny_applied: Option<u64>) -> Snapshot {
|
||||
Snapshot {
|
||||
readiness: crate::warnings::readiness(),
|
||||
capabilities: &[
|
||||
swarm_wire::hive_status::TOLERANT_PERMISSIONS,
|
||||
swarm_wire::hive_status::NO_CONFIG_CHECKOUT,
|
||||
],
|
||||
egress_deny_applied,
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -404,7 +419,7 @@ async fn publish(client: &async_nats::Client, hive: &str) -> Result<()> {
|
|||
|
||||
#[cfg(test)]
|
||||
mod snapshot_tests {
|
||||
use super::snapshot;
|
||||
use super::{snapshot, snapshot_with};
|
||||
|
||||
/// The controller reads the capability out of the published bytes with
|
||||
/// the wire type, and the readiness keeps its own top-level fields.
|
||||
|
|
@ -421,4 +436,22 @@ mod snapshot_tests {
|
|||
);
|
||||
assert!(value.get("status").is_some() && value.get("warnings").is_some());
|
||||
}
|
||||
|
||||
/// Before a pass has rendered the list the field is absent, as on a hive that
|
||||
/// predates it; after, the controller reads the revision with the wire type.
|
||||
#[test]
|
||||
fn the_snapshot_carries_the_applied_deny_revision_once_there_is_one() {
|
||||
let fresh = serde_json::to_value(snapshot_with(None)).unwrap();
|
||||
assert!(
|
||||
fresh
|
||||
.get(swarm_wire::hive_status::EGRESS_DENY_APPLIED)
|
||||
.is_none()
|
||||
);
|
||||
assert_eq!(swarm_wire::hive_status::egress_deny_applied(&fresh), None);
|
||||
let rendered = serde_json::to_value(snapshot_with(Some(4))).unwrap();
|
||||
assert_eq!(
|
||||
swarm_wire::hive_status::egress_deny_applied(&rendered),
|
||||
Some(4)
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -12,6 +12,7 @@ pub mod knowledge;
|
|||
pub mod mcp_sockets;
|
||||
pub mod permissions_import;
|
||||
pub mod squid_allow;
|
||||
pub mod squid_deny;
|
||||
pub mod squid_users;
|
||||
pub mod swarm_limits;
|
||||
pub mod swarm_permissions;
|
||||
|
|
|
|||
|
|
@ -294,7 +294,7 @@ fn seeded(seed: Option<&Path>, logging: &[String]) -> Result<Rendered> {
|
|||
}
|
||||
|
||||
/// `names` sorted, with each one that another covers dropped.
|
||||
fn reduced(mut names: Vec<String>) -> Vec<String> {
|
||||
pub(super) fn reduced(mut names: Vec<String>) -> Vec<String> {
|
||||
names.sort();
|
||||
names.dedup();
|
||||
let all = names.clone();
|
||||
|
|
|
|||
452
hive-c0re/src/workers/squid_deny.rs
Normal file
452
hive-c0re/src/workers/squid_deny.rs
Normal file
|
|
@ -0,0 +1,452 @@
|
|||
//! Squid's deny-list file: the destination names the swarm refuses this hive's
|
||||
//! agents, whatever allows them.
|
||||
//!
|
||||
//! swarm-controller keeps the deny list beside the allow-list
|
||||
//! ([`swarm_wire::egress_allow::DenyList`], its own key of the same bucket); each
|
||||
//! entry is for every agent or for one. Every [`INTERVAL`] this reads it and
|
||||
//! renders [`ENV_DENY_FILE`], squid configuration that
|
||||
//! `nix/host-modules/hive-squid.nix` `include`s after the destination denies
|
||||
//! and before `deny !authed` and every allow, so a denied name is refused for
|
||||
//! every user in every mode. A swarm-wide entry is one `dstdomain` ACL and a
|
||||
//! `deny` that names no user, so even a client without a credential gets a 403;
|
||||
//! an agent's entry also names that agent's squid user. Each rule ends in the
|
||||
//! [`MARK`] annotation, which the proxy logs as its own `egress_denied` line.
|
||||
//!
|
||||
//! No entry renders an empty file, which squid reads as no rules, so a hive
|
||||
//! whose swarm has no deny list configures squid as it did before this existed.
|
||||
//!
|
||||
//! The revision of the list last rendered is what [`applied`] answers and the
|
||||
//! hive's status snapshot carries, so the swarm can see a hive that is behind.
|
||||
//! A hive that cannot read the list keeps the file it has, and reports nothing
|
||||
//! it did not render.
|
||||
//!
|
||||
//! Every entry is checked again with
|
||||
//! [`swarm_wire::egress_allow::normalize_domain`], since the names end up in
|
||||
//! squid's configuration, and skipped when it fails. An entry another entry of
|
||||
//! its list covers is dropped, because squid ignores the narrower of two
|
||||
//! overlapping names in one ACL.
|
||||
|
||||
use std::fmt::Write as _;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use std::time::Duration;
|
||||
|
||||
use anyhow::{Context, Result};
|
||||
use swarm_queue_client::egress_allow::{self, DenyRead};
|
||||
use swarm_wire::egress_allow::{DenyList, covered_by, normalize_domain};
|
||||
|
||||
use super::squid_allow::reduced;
|
||||
use super::squid_users::{read_or_empty, write_if_changed};
|
||||
use crate::stats::sweep_health::{self, SweepHealth};
|
||||
|
||||
/// How often the list is read.
|
||||
pub const INTERVAL: Duration = Duration::from_secs(30);
|
||||
|
||||
/// At [`INTERVAL`] this is ~90 seconds of failing reads.
|
||||
const FAILURES_BEFORE_WARNING: u32 = 3;
|
||||
|
||||
/// The squid configuration file this renders. Set by `nix/host-modules/hive-c0re`
|
||||
/// from `services.hyperhive.deploy.hive-controller.squid.denyFile`.
|
||||
pub const ENV_DENY_FILE: &str = "HIVE_SQUID_DENY_FILE";
|
||||
|
||||
/// The annotation a deny rule puts on a request, as `key=value`. The `note` ACL
|
||||
/// in `nix/host-modules/hive-squid.nix` reads the key.
|
||||
const MARK: &str = "egress_deny=1";
|
||||
|
||||
/// The ACL that applies [`MARK`].
|
||||
const MARK_ACL: &str = "deny_hit_mark";
|
||||
|
||||
/// The revision last rendered plus one; `0` before the first pass has rendered
|
||||
/// or confirmed anything.
|
||||
static APPLIED: AtomicU64 = AtomicU64::new(0);
|
||||
|
||||
/// The revision of the swarm's deny list this hive's file holds, `0` when the
|
||||
/// swarm has none, or `None` until a pass has said so.
|
||||
#[must_use]
|
||||
pub fn applied() -> Option<u64> {
|
||||
APPLIED.load(Ordering::Relaxed).checked_sub(1)
|
||||
}
|
||||
|
||||
fn set_applied(revision: u64) {
|
||||
APPLIED.store(revision.saturating_add(1), Ordering::Relaxed);
|
||||
}
|
||||
|
||||
/// Read the list now and every [`INTERVAL`] until shutdown, when this host has
|
||||
/// squid's deny file.
|
||||
pub async fn follow(client: async_nats::Client, mut shutdown: tokio::sync::watch::Receiver<bool>) {
|
||||
let Some(file) = std::env::var_os(ENV_DENY_FILE)
|
||||
.filter(|v| !v.is_empty())
|
||||
.map(PathBuf::from)
|
||||
else {
|
||||
return;
|
||||
};
|
||||
let mut health = SweepHealth::new("squid_deny_render", "warn", FAILURES_BEFORE_WARNING);
|
||||
loop {
|
||||
match pass(&client, &file).await {
|
||||
Ok(()) => health.record_ok(),
|
||||
Err(e) => {
|
||||
tracing::warn!(error = ?e, "squid deny list: the pass failed");
|
||||
let err = format!("{e:#}");
|
||||
health.record_err(|ctx| {
|
||||
let age = ctx.since_last_ok.map_or_else(
|
||||
|| "no success this session".to_owned(),
|
||||
|d| format!("last ok {} ago", sweep_health::fmt_age(d)),
|
||||
);
|
||||
format!(
|
||||
"reading the swarm's egress deny list is failing ({} consecutive, \
|
||||
{age}) — squid keeps the denies last written: {err}",
|
||||
ctx.consecutive
|
||||
)
|
||||
});
|
||||
}
|
||||
}
|
||||
tokio::select! {
|
||||
() = tokio::time::sleep(INTERVAL) => {}
|
||||
_ = shutdown.changed() => return,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async fn pass(client: &async_nats::Client, file: &Path) -> Result<()> {
|
||||
// An unconnected client does not fail a JetStream request, it hangs on it.
|
||||
swarm_queue_client::ensure_connected(client)?;
|
||||
let agents: Vec<String> = crate::lifecycle::agents_for_meta_listing()
|
||||
.await
|
||||
.context("enumerating this hive's agents")?
|
||||
.into_iter()
|
||||
.map(|spec| spec.name)
|
||||
.collect();
|
||||
let read = egress_allow::read_deny(client)
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!(e))
|
||||
.context("reading the swarm's egress deny list")?;
|
||||
if let Some(revision) = sync_with(read, &agents, file)? {
|
||||
set_applied(revision);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// [`pass`] with the read and the agents given. The file is only written when
|
||||
/// its rendering differs from what it holds. Answers the revision the file now
|
||||
/// holds, or `None` when it cannot say.
|
||||
fn sync_with(read: DenyRead, agents: &[String], file: &Path) -> Result<Option<u64>> {
|
||||
match read {
|
||||
DenyRead::Current { list, revision } => {
|
||||
let rendered = render(&list, agents);
|
||||
if write_if_changed(file, &rendered.text)? {
|
||||
tracing::info!(
|
||||
file = %file.display(), revision,
|
||||
"squid deny list: {} swarm, {} agent entries", rendered.swarm, rendered.agent
|
||||
);
|
||||
}
|
||||
if !rendered.skipped.is_empty() {
|
||||
tracing::warn!(
|
||||
skipped = ?rendered.skipped,
|
||||
"squid deny list: entries the swarm holds were skipped as invalid"
|
||||
);
|
||||
}
|
||||
Ok(Some(revision))
|
||||
}
|
||||
// Nothing to read says nothing about what the swarm wants, so a file
|
||||
// that holds denies stays. An empty file is the true "no list".
|
||||
DenyRead::NotSet => Ok(read_or_empty(file)?.is_empty().then_some(0)),
|
||||
}
|
||||
}
|
||||
|
||||
struct Rendered {
|
||||
text: String,
|
||||
/// Swarm-wide names written.
|
||||
swarm: usize,
|
||||
/// Names written across this hive's agents.
|
||||
agent: usize,
|
||||
/// Entries left out because they are not valid names or agents.
|
||||
skipped: Vec<String>,
|
||||
}
|
||||
|
||||
/// The squid configuration for `list`, for the agents this hive runs. Empty for
|
||||
/// no entry that applies.
|
||||
///
|
||||
/// The destination ACL comes first in each rule: squid checks ACLs left to
|
||||
/// right, so a swarm-wide rule never reaches a `proxy_auth` ACL, which would
|
||||
/// answer an anonymous client 407 instead of the 403.
|
||||
fn render(list: &DenyList, agents: &[String]) -> Rendered {
|
||||
let mut skipped = Vec::new();
|
||||
let mut swarm_wide = Vec::new();
|
||||
let mut per_agent: std::collections::BTreeMap<&str, Vec<String>> =
|
||||
std::collections::BTreeMap::new();
|
||||
for entry in &list.entries {
|
||||
let Ok(domain) = normalize_domain(&entry.domain) else {
|
||||
skipped.push(entry.domain.clone());
|
||||
continue;
|
||||
};
|
||||
match entry.agent.as_deref() {
|
||||
None => swarm_wide.push(domain),
|
||||
Some(agent) => {
|
||||
if hive_types::Ident::parse(agent).is_err() {
|
||||
skipped.push(format!("{} for {agent}", entry.domain));
|
||||
} else if let Some(mine) = agents.iter().find(|a| *a == agent) {
|
||||
per_agent.entry(mine.as_str()).or_default().push(domain);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
let swarm_wide = reduced(swarm_wide);
|
||||
|
||||
let mut rules = String::new();
|
||||
if !swarm_wide.is_empty() {
|
||||
let _ = write!(
|
||||
rules,
|
||||
"acl deny_swarm_dst dstdomain {}\n\
|
||||
http_access deny deny_swarm_dst {MARK_ACL}\n",
|
||||
swarm_wide.join(" ")
|
||||
);
|
||||
}
|
||||
let mut agent_count = 0;
|
||||
for (agent, names) in per_agent {
|
||||
let names = reduced(
|
||||
names
|
||||
.into_iter()
|
||||
.filter(|d| !covered_by(d, &swarm_wide))
|
||||
.collect(),
|
||||
);
|
||||
if names.is_empty() {
|
||||
continue;
|
||||
}
|
||||
agent_count += names.len();
|
||||
let _ = write!(
|
||||
rules,
|
||||
"acl deny_agent_{agent}_dst dstdomain {}\n\
|
||||
acl deny_agent_{agent} proxy_auth {agent}\n\
|
||||
http_access deny deny_agent_{agent}_dst deny_agent_{agent} {MARK_ACL}\n",
|
||||
names.join(" ")
|
||||
);
|
||||
}
|
||||
let text = if rules.is_empty() {
|
||||
String::new()
|
||||
} else {
|
||||
format!("acl {MARK_ACL} annotate_transaction {MARK}\n{rules}")
|
||||
};
|
||||
Rendered {
|
||||
text,
|
||||
swarm: swarm_wide.len(),
|
||||
agent: agent_count,
|
||||
skipped,
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use swarm_wire::egress_allow::DenyEntry;
|
||||
|
||||
fn list(entries: &[(&str, Option<&str>)]) -> DenyList {
|
||||
DenyList {
|
||||
entries: entries
|
||||
.iter()
|
||||
.map(|(d, a)| DenyEntry {
|
||||
domain: (*d).to_owned(),
|
||||
agent: a.map(str::to_owned),
|
||||
})
|
||||
.collect(),
|
||||
}
|
||||
}
|
||||
|
||||
fn agents(names: &[&str]) -> Vec<String> {
|
||||
names.iter().map(|n| (*n).to_owned()).collect()
|
||||
}
|
||||
|
||||
const MARK_LINE: &str = "acl deny_hit_mark annotate_transaction egress_deny=1\n";
|
||||
|
||||
/// The one place a rule's shape is pinned: the destination ACL before any
|
||||
/// user ACL, and the annotation last.
|
||||
#[test]
|
||||
fn a_swarm_wide_name_is_one_destination_rule_that_names_no_user() {
|
||||
let r = render(
|
||||
&list(&[("evil.com", None), (".bad.io", None), ("EVIL.com", None)]),
|
||||
&agents(&["atlas"]),
|
||||
);
|
||||
assert_eq!(
|
||||
r.text,
|
||||
format!(
|
||||
"{MARK_LINE}acl deny_swarm_dst dstdomain .bad.io evil.com\n\
|
||||
http_access deny deny_swarm_dst deny_hit_mark\n"
|
||||
)
|
||||
);
|
||||
assert!(!r.text.contains("proxy_auth"), "no user, so no 407");
|
||||
assert_eq!((r.swarm, r.agent), (2, 0));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_agents_name_is_a_destination_then_user_rule_and_other_hives_agents_are_left_out() {
|
||||
let r = render(
|
||||
&list(&[
|
||||
("pypi.org", Some("atlas")),
|
||||
("crates.io", Some("atlas")),
|
||||
("github.com", Some("iris")),
|
||||
("github.com", Some("../argus")),
|
||||
]),
|
||||
&agents(&["atlas", "argus"]),
|
||||
);
|
||||
assert_eq!(
|
||||
r.text,
|
||||
format!(
|
||||
"{MARK_LINE}acl deny_agent_atlas_dst dstdomain crates.io pypi.org\n\
|
||||
acl deny_agent_atlas proxy_auth atlas\n\
|
||||
http_access deny deny_agent_atlas_dst deny_agent_atlas deny_hit_mark\n"
|
||||
)
|
||||
);
|
||||
assert_eq!((r.swarm, r.agent), (0, 2));
|
||||
assert_eq!(r.skipped, ["github.com for ../argus"]);
|
||||
assert!(!r.text.contains("iris"), "another hive's agent");
|
||||
assert!(!r.text.contains("argus"), "an invalid name");
|
||||
}
|
||||
|
||||
/// An agent's entry adds to the swarm-wide rule and never replaces it: the
|
||||
/// swarm-wide rule renders whole whatever the agent lists, and a name it
|
||||
/// already covers is not repeated for the agent.
|
||||
#[test]
|
||||
fn an_agents_entry_adds_to_the_swarm_wide_rule_and_never_narrows_it() {
|
||||
let r = render(
|
||||
&list(&[
|
||||
(".evil.com", None),
|
||||
("api.evil.com", Some("atlas")),
|
||||
("bad.io", Some("atlas")),
|
||||
]),
|
||||
&agents(&["atlas"]),
|
||||
);
|
||||
assert!(
|
||||
r.text.contains(
|
||||
"acl deny_swarm_dst dstdomain .evil.com\nhttp_access deny deny_swarm_dst"
|
||||
)
|
||||
);
|
||||
assert!(r.text.contains("deny_agent_atlas_dst dstdomain bad.io\n"));
|
||||
assert!(!r.text.contains("api.evil.com"), "{}", r.text);
|
||||
assert_eq!((r.swarm, r.agent), (1, 1));
|
||||
}
|
||||
|
||||
/// No entry renders no byte, which is what keeps a hive with no deny list
|
||||
/// configured as it was.
|
||||
#[test]
|
||||
fn no_entry_that_applies_renders_an_empty_file() {
|
||||
assert_eq!(render(&DenyList::default(), &agents(&["atlas"])).text, "");
|
||||
// An agent this hive does not run, and an invalid name, render nothing.
|
||||
let r = render(
|
||||
&list(&[("evil.com", Some("iris")), ("1.2.3.4", None)]),
|
||||
&agents(&["atlas"]),
|
||||
);
|
||||
assert_eq!(r.text, "");
|
||||
assert_eq!(r.skipped, ["1.2.3.4"]);
|
||||
// Control: the same list with a name that applies is not empty.
|
||||
assert!(
|
||||
!render(&list(&[("evil.com", None)]), &agents(&[]))
|
||||
.text
|
||||
.is_empty()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_entry_that_is_not_a_name_never_reaches_squid_configuration() {
|
||||
let r = render(
|
||||
&list(&[
|
||||
("evil.com", None),
|
||||
("*.example.com", Some("atlas")),
|
||||
("example.com\nhttp_access allow all", Some("atlas")),
|
||||
("example.com\nhttp_access allow all", None),
|
||||
]),
|
||||
&agents(&["atlas"]),
|
||||
);
|
||||
assert!(!r.text.contains("allow"), "{}", r.text);
|
||||
assert!(r.text.contains("dstdomain evil.com\n"));
|
||||
assert_eq!(r.skipped.len(), 3);
|
||||
}
|
||||
|
||||
/// The value swarm-controller's `egress_deny` route publishes, byte for byte
|
||||
/// as its own test pins it, renders the rules for the agents named in it.
|
||||
#[test]
|
||||
fn the_controllers_published_value_renders_the_rules_it_names() {
|
||||
let wire = r#"{"entries":[{"domain":"bad.io","agent":"atlas"},{"domain":"evil.com"},{"domain":"evil.com","agent":"atlas"}]}"#;
|
||||
let decoded: DenyList = serde_json::from_str(wire).expect("decodes");
|
||||
let r = render(&decoded, &agents(&["atlas"]));
|
||||
assert!(r.text.contains("acl deny_swarm_dst dstdomain evil.com\n"));
|
||||
assert!(
|
||||
r.text
|
||||
.contains("acl deny_agent_atlas_dst dstdomain bad.io\n")
|
||||
);
|
||||
assert_eq!((r.swarm, r.agent), (1, 1));
|
||||
}
|
||||
|
||||
struct Dir(tempfile::TempDir);
|
||||
|
||||
impl Dir {
|
||||
fn file(&self) -> PathBuf {
|
||||
self.0.path().join("deny.conf")
|
||||
}
|
||||
}
|
||||
|
||||
fn dir() -> Dir {
|
||||
Dir(tempfile::tempdir().expect("tempdir"))
|
||||
}
|
||||
|
||||
fn text(d: &Dir) -> String {
|
||||
std::fs::read_to_string(d.file()).expect("the file is there")
|
||||
}
|
||||
|
||||
fn current(entries: &[(&str, Option<&str>)], revision: u64) -> DenyRead {
|
||||
DenyRead::Current {
|
||||
list: list(entries),
|
||||
revision,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_file_follows_the_list_and_the_claimed_revision_follows_the_file() {
|
||||
let d = dir();
|
||||
let mine = agents(&["atlas"]);
|
||||
let sync = |read| sync_with(read, &mine, &d.file()).expect("renders");
|
||||
|
||||
assert_eq!(sync(current(&[("evil.com", None)], 7)), Some(7));
|
||||
assert!(text(&d).contains("dstdomain evil.com"));
|
||||
|
||||
// A second pass of the same list writes nothing.
|
||||
let ino = std::os::unix::fs::MetadataExt::ino(&std::fs::metadata(d.file()).unwrap());
|
||||
assert_eq!(sync(current(&[("evil.com", None)], 7)), Some(7));
|
||||
assert_eq!(
|
||||
std::os::unix::fs::MetadataExt::ino(&std::fs::metadata(d.file()).unwrap()),
|
||||
ino
|
||||
);
|
||||
|
||||
// No answer keeps the denies in force and claims no revision.
|
||||
assert_eq!(sync(DenyRead::NotSet), None);
|
||||
assert!(text(&d).contains("dstdomain evil.com"), "denies are kept");
|
||||
|
||||
// The list emptied: the file is empty and the revision moves.
|
||||
assert_eq!(sync(current(&[], 9)), Some(9));
|
||||
assert_eq!(text(&d), "");
|
||||
|
||||
// An empty file with nothing to read is the true "no list": revision 0.
|
||||
assert_eq!(sync(DenyRead::NotSet), Some(0));
|
||||
}
|
||||
|
||||
/// The only test that touches the process-wide value.
|
||||
#[test]
|
||||
fn nothing_is_applied_until_a_pass_says_so() {
|
||||
assert_eq!(applied(), None);
|
||||
set_applied(0);
|
||||
assert_eq!(applied(), Some(0));
|
||||
set_applied(12);
|
||||
assert_eq!(applied(), Some(12));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_file_is_group_readable() {
|
||||
use std::os::unix::fs::PermissionsExt as _;
|
||||
let d = dir();
|
||||
sync_with(current(&[("evil.com", None)], 1), &agents(&[]), &d.file()).expect("renders");
|
||||
let mode = std::fs::metadata(d.file())
|
||||
.expect("stat")
|
||||
.permissions()
|
||||
.mode()
|
||||
& 0o777;
|
||||
assert_eq!(mode, 0o640);
|
||||
}
|
||||
}
|
||||
|
|
@ -299,6 +299,9 @@ in
|
|||
HIVE_SQUID_ALLOWED_SEED_FILE =
|
||||
config.services.hyperhive.deploy.hive-controller.squid.allowedSeedFile;
|
||||
HIVE_SQUID_ALWAYS_ALLOWED = lib.concatStringsSep " " config.services.hyperhive.deploy.hive-controller.squid.alwaysAllowed;
|
||||
# The swarm's egress deny list, which `workers::squid_deny` renders into the
|
||||
# include squid reads before every allow.
|
||||
HIVE_SQUID_DENY_FILE = config.services.hyperhive.deploy.hive-controller.squid.denyFile;
|
||||
# The egress proxy facts every container is handed (`HiveFacts` in
|
||||
# `hive-priv-sock`). They sit with the users file above because that is what
|
||||
# makes squid's credential check and per-agent policy real on this host: an
|
||||
|
|
|
|||
|
|
@ -10,6 +10,8 @@
|
|||
# the swarm lists for that agent alone. A third kind, set per agent
|
||||
# (`internet_log`), reaches the same names but is also let through to any other
|
||||
# destination the deny rules leave; the request is logged as `would_deny`.
|
||||
# The swarm's deny list (rendered by hive-c0re into `denyFile`) is refused for
|
||||
# all three before any allow, and logged as `egress_denied`.
|
||||
# An agent sends its credential
|
||||
# once it has fetched it (../agent-modules/egress-proxy.nix); the exporter,
|
||||
# journal forwarding and the self-check below show the proxy works.
|
||||
|
|
@ -89,6 +91,10 @@ let
|
|||
# `deny !allowed_dst`, so it sees the same denies and only turns that last
|
||||
# refusal into an allow. The `dst` denies cost an unlisted name a lookup on
|
||||
# the host's resolver before it is refused.
|
||||
# The swarm's deny list (`denyFile`) sits right after those destination
|
||||
# denies and before `deny !authed`, so a denied name is refused before any
|
||||
# credential check and before every allow, in every mode: its rules name the
|
||||
# destination first, and a swarm-wide one names no user at all.
|
||||
squidConf = ''
|
||||
http_port ${listen}
|
||||
pid_filename /run/squid/squid.pid
|
||||
|
|
@ -110,6 +116,14 @@ let
|
|||
acl would_denied note would_deny 1
|
||||
access_log syslog:daemon.info logformat=would_deny would_denied
|
||||
|
||||
# A request one of the deny list's rules (hive-c0re's `denyFile`, below)
|
||||
# refused carries the annotation `egress_deny=1` and gets a line of its own,
|
||||
# distinct from `would_deny` and from the 403 line every refusal has, in
|
||||
# every mode.
|
||||
logformat egress_denied egress_denied user=%[un method=%rm host=%>rd port=%>rP result=%Ss/%03>Hs
|
||||
acl egress_denied_hit note egress_deny 1
|
||||
access_log syslog:daemon.info logformat=egress_denied egress_denied_hit
|
||||
|
||||
# The helper reads the users file on every request and reloads it when its
|
||||
# mtime moves. It exits when the file is missing, which is why the file is
|
||||
# created before squid starts. A positive answer is cached for a minute, so
|
||||
|
|
@ -153,6 +167,7 @@ let
|
|||
http_access deny to_linklocal
|
||||
http_access deny to_bridge
|
||||
http_access deny to_private
|
||||
include ${cfg.denyFile}
|
||||
http_access deny !authed
|
||||
${lib.optionalString (base != [ ]) "http_access allow base_dst"}
|
||||
include ${cfg.agentAllowFile}
|
||||
|
|
@ -162,9 +177,9 @@ let
|
|||
http_access deny all
|
||||
'';
|
||||
|
||||
# `squid -k parse` on the configuration with the include pointing at an empty
|
||||
# `squid -k parse` on the configuration with the includes pointing at an empty
|
||||
# file. The nixpkgs module runs the same parse on the real text, which
|
||||
# cannot open the include in the build sandbox.
|
||||
# cannot open the includes in the build sandbox.
|
||||
squidConfCheck =
|
||||
pkgs.runCommand "squid-conf-check"
|
||||
{
|
||||
|
|
@ -173,7 +188,7 @@ let
|
|||
''
|
||||
squid -k parse -f ${
|
||||
pkgs.writeText "squid.conf" (
|
||||
builtins.replaceStrings [ cfg.agentAllowFile ] [ "/dev/null" ] squidConf
|
||||
builtins.replaceStrings [ cfg.agentAllowFile cfg.denyFile ] [ "/dev/null" "/dev/null" ] squidConf
|
||||
)
|
||||
}
|
||||
touch $out
|
||||
|
|
@ -381,6 +396,24 @@ in
|
|||
'';
|
||||
};
|
||||
|
||||
denyFile = lib.mkOption {
|
||||
type = lib.types.path;
|
||||
default = "${builtins.dirOf cfg.usersFile}/deny.conf";
|
||||
readOnly = true;
|
||||
defaultText = lib.literalMD "`deny.conf`, in the directory of `usersFile`";
|
||||
description = ''
|
||||
Squid configuration that hive-c0re generates from the swarm's deny list
|
||||
(`/api/egress/denied-domains`) and the main configuration includes after
|
||||
the destination denies and before `deny !authed` and every allow: a
|
||||
`http_access deny` for the swarm-wide names, and one per agent of this
|
||||
hive the swarm lists names for. A denied name is refused for every user
|
||||
in every egress mode, whether or not an allow rule covers it. The file
|
||||
is empty, and squid configured as without it, until the swarm lists a
|
||||
name. Squid is reloaded when it changes. Owned by `hive-core` and readable
|
||||
by the `squid` group.
|
||||
'';
|
||||
};
|
||||
|
||||
allowedSeedFile = lib.mkOption {
|
||||
type = lib.types.path;
|
||||
default = seedFile;
|
||||
|
|
@ -481,6 +514,7 @@ in
|
|||
"f ${cfg.unrestrictedFile} 0640 hive-core squid - -"
|
||||
"C ${cfg.allowedFile} 0640 hive-core squid - ${seedFile}"
|
||||
"f ${cfg.agentAllowFile} 0640 hive-core squid - -"
|
||||
"f ${cfg.denyFile} 0640 hive-core squid - -"
|
||||
];
|
||||
|
||||
# Squid reads the unrestricted list, the allow-list and the per-agent include
|
||||
|
|
@ -494,6 +528,7 @@ in
|
|||
cfg.unrestrictedFile
|
||||
cfg.allowedFile
|
||||
cfg.agentAllowFile
|
||||
cfg.denyFile
|
||||
];
|
||||
};
|
||||
|
||||
|
|
|
|||
|
|
@ -636,6 +636,58 @@ let
|
|||
&& builtins.elem ''acl unrestricted proxy_auth "${list}"'' lines
|
||||
&& list == "/var/lib/hive-squid/unrestricted";
|
||||
}
|
||||
{
|
||||
# The swarm's deny list is an include that precedes every allow and the
|
||||
# credential check, so a denied name is refused in every mode and for a
|
||||
# client that sent no credential. Control: the include and the allows are
|
||||
# found, so a position compared below is not a lookup of nothing.
|
||||
name = "squid includes the swarm's deny list after the destination denies and before the credential check and every allow";
|
||||
ok =
|
||||
let
|
||||
sq = hiveOtel.services.hyperhive.deploy.hive-controller.squid;
|
||||
lines = lib.splitString "\n" hiveOtel.services.squid.configText;
|
||||
at = l: lib.lists.findFirstIndex (x: x == l) null lines;
|
||||
deny = at "include ${sq.denyFile}";
|
||||
allows = map at [
|
||||
"http_access allow base_dst"
|
||||
"include ${sq.agentAllowFile}"
|
||||
"http_access allow unrestricted"
|
||||
"http_access allow allowed_dst"
|
||||
];
|
||||
in
|
||||
deny != null
|
||||
&& sq.denyFile == "/var/lib/hive-squid/deny.conf"
|
||||
&& !(lib.elem null allows)
|
||||
&& lib.all (a: deny < a) allows
|
||||
&& at "http_access deny to_private" < deny
|
||||
&& at "http_access deny ip_v4_host" < deny
|
||||
&& deny < at "http_access deny !authed"
|
||||
# Two includes: this one and the per-agent allow rules.
|
||||
&& lib.length (lib.filter (l: lib.hasPrefix "include " l) lines) == 2;
|
||||
}
|
||||
{
|
||||
# What the deny list adds to the configuration is the include and the
|
||||
# lines that log a hit; with the file empty, which tmpfiles creates, squid
|
||||
# has no rule it did not have. Control: the lines are found, so the
|
||||
# equality below is not against an empty filter.
|
||||
name = "squid's deny list adds one include and the log lines, and its file exists empty before squid starts";
|
||||
ok =
|
||||
let
|
||||
sq = hiveOtel.services.hyperhive.deploy.hive-controller.squid;
|
||||
conf = hiveOtel.services.squid.configText;
|
||||
mentions = lib.filter (
|
||||
l: !(lib.hasPrefix "#" l) && (lib.hasInfix "egress_den" l || lib.hasInfix sq.denyFile l)
|
||||
) (lib.splitString "\n" conf);
|
||||
in
|
||||
mentions == [
|
||||
"logformat egress_denied egress_denied user=%[un method=%rm host=%>rd port=%>rP result=%Ss/%03>Hs"
|
||||
"acl egress_denied_hit note egress_deny 1"
|
||||
"access_log syslog:daemon.info logformat=egress_denied egress_denied_hit"
|
||||
"include ${sq.denyFile}"
|
||||
]
|
||||
&& builtins.elem "f ${sq.denyFile} 0640 hive-core squid - -" hiveOtel.systemd.tmpfiles.rules
|
||||
&& hiveOtelAllLocal.systemd.services.hive-c0re.environment.HIVE_SQUID_DENY_FILE == sq.denyFile;
|
||||
}
|
||||
{
|
||||
# The list is read when squid parses its configuration, so a change to
|
||||
# the file reloads squid. Control: the watched path is the list's own.
|
||||
|
|
@ -651,6 +703,7 @@ let
|
|||
list
|
||||
hiveOtel.services.hyperhive.deploy.hive-controller.squid.allowedFile
|
||||
hiveOtel.services.hyperhive.deploy.hive-controller.squid.agentAllowFile
|
||||
hiveOtel.services.hyperhive.deploy.hive-controller.squid.denyFile
|
||||
]
|
||||
&& hiveOtel.systemd.services.squid-unrestricted-reload.startLimitIntervalSec == 0
|
||||
&& builtins.elem "paths.target" path.wantedBy
|
||||
|
|
|
|||
|
|
@ -253,6 +253,7 @@ pkgs.testers.runNixOSTest {
|
|||
LIST = "/var/lib/hive-squid/unrestricted"
|
||||
ALLOWED = "/var/lib/hive-squid/allowed"
|
||||
AGENT_ALLOW = "/var/lib/hive-squid/agent-allow.conf"
|
||||
DENY = "/var/lib/hive-squid/deny.conf"
|
||||
UNRESTRICTED = ["uma", "selfcheck-unrestricted.squid"]
|
||||
|
||||
start_all()
|
||||
|
|
@ -304,6 +305,14 @@ pkgs.testers.runNixOSTest {
|
|||
f"&& chmod 640 {path}.new && mv {path}.new {path}"
|
||||
)
|
||||
|
||||
def write_deny(body):
|
||||
# What hive-c0re does for the deny list: written aside and renamed into
|
||||
# place, in squid's group; the reload is squid-unrestricted-reload's.
|
||||
proxy.succeed(
|
||||
f"printf '%s' {shlex.quote(body)} > {DENY}.new && chown hive-core:squid {DENY}.new "
|
||||
f"&& chmod 640 {DENY}.new && mv {DENY}.new {DENY}"
|
||||
)
|
||||
|
||||
def wait_status(host, cred, want):
|
||||
# The reload that follows a list write is asynchronous.
|
||||
got = None
|
||||
|
|
@ -328,6 +337,8 @@ pkgs.testers.runNixOSTest {
|
|||
assert proxy.succeed(f"stat -c '%U:%G %a' {ALLOWED}").strip() == "hive-core:squid 640"
|
||||
assert proxy.succeed(f"stat -c '%U:%G %a' {AGENT_ALLOW}").strip() == "hive-core:squid 640"
|
||||
assert proxy.succeed(f"stat -c %s {AGENT_ALLOW}").strip() == "0"
|
||||
assert proxy.succeed(f"stat -c '%U:%G %a' {DENY}").strip() == "hive-core:squid 640"
|
||||
assert proxy.succeed(f"stat -c %s {DENY}").strip() == "0", "no deny list, no rule"
|
||||
# The allow-list file starts as the hive's own allowedDomains.
|
||||
proxy.succeed(f"grep -Fx forge.t.local {ALLOWED}")
|
||||
assert connect("forge.t.local", ATLAS) == "407", "no such user"
|
||||
|
|
@ -573,6 +584,53 @@ pkgs.testers.runNixOSTest {
|
|||
write_allow(seeded, "")
|
||||
wait_status("other.t.local", RUTH, "403")
|
||||
|
||||
with subtest("the swarm's deny list is refused before any allow, in every mode, with or without a credential, and logged as egress_denied"):
|
||||
# What hive-c0re renders: `forge.t.local` and `other.t.local` for every
|
||||
# agent, `api.t.local` (a base-list name) for atlas alone. `ruth` is in
|
||||
# log mode, so her rule follows. The names are allowed or reachable
|
||||
# without the list, which the controls establish first.
|
||||
assert connect("forge.t.local", ATLAS) == "200", "control: allowed by the hive"
|
||||
assert connect("forge.t.local", UMA) == "200", "control: allowed, unrestricted"
|
||||
assert connect("other.t.local", UMA) == "200", "control: unrestricted reaches it"
|
||||
assert connect("api.t.local", ATLAS) == "200", "control: base list"
|
||||
write_allow(
|
||||
seeded,
|
||||
"# source: swarm\n"
|
||||
"acl log_mark annotate_transaction would_deny=1\n"
|
||||
"acl agent_log_ruth proxy_auth ruth\n"
|
||||
"http_access allow agent_log_ruth !allowed_dst log_mark\n",
|
||||
)
|
||||
wait_status("other.t.local", RUTH, "200")
|
||||
write_deny(
|
||||
"acl deny_hit_mark annotate_transaction egress_deny=1\n"
|
||||
"acl deny_swarm_dst dstdomain forge.t.local other.t.local\n"
|
||||
"http_access deny deny_swarm_dst deny_hit_mark\n"
|
||||
"acl deny_agent_atlas_dst dstdomain api.t.local\n"
|
||||
"acl deny_agent_atlas proxy_auth atlas\n"
|
||||
"http_access deny deny_agent_atlas_dst deny_agent_atlas deny_hit_mark\n"
|
||||
)
|
||||
wait_status("forge.t.local", ATLAS, "403")
|
||||
assert connect("forge.t.local", RUTH) == "403", "a listed name, log mode"
|
||||
assert connect("forge.t.local", UMA) == "403", "a listed name, unrestricted"
|
||||
assert connect("other.t.local", UMA) == "403", "unrestricted"
|
||||
assert connect("other.t.local", RUTH) == "403", "log mode"
|
||||
assert connect("forge.t.local") == "403", "no credential: refused, not challenged"
|
||||
assert connect("api.t.local", ATLAS) == "403", "a base-list name, denied for atlas"
|
||||
assert connect("api.t.local", RUTH) == "200", "control: another agent keeps the base name"
|
||||
assert connect("api.t.local") == "407", "control: a per-agent deny does not answer a client it cannot name"
|
||||
proxy.wait_until_succeeds(
|
||||
"journalctl -u squid.service -o cat | grep -E "
|
||||
"'egress_denied user=atlas method=CONNECT host=api\\.t\\.local port=443 result=TCP_DENIED/403'"
|
||||
)
|
||||
assert squid_saw(r"egress_denied user=- method=CONNECT host=forge\.t\.local port=443 "), "the anonymous client's hit"
|
||||
assert not squid_saw(r"egress_denied .* host=loop\.t\.local"), "another refusal is not a deny-list hit"
|
||||
assert not squid_saw(r"egress_denied user=ruth .* host=api\.t\.local"), "another agent's name is not a hit"
|
||||
write_deny("")
|
||||
write_allow(seeded, "")
|
||||
wait_status("forge.t.local", ATLAS, "200")
|
||||
assert connect("api.t.local", ATLAS) == "200", "the list emptied"
|
||||
assert connect("other.t.local", UMA) == "200", "the list emptied, unrestricted"
|
||||
|
||||
with subtest("squid starts on a missing list, treats every user as restricted, and recovers when it appears"):
|
||||
proxy.succeed(f"rm {LIST}")
|
||||
proxy.succeed("systemctl restart squid.service")
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
//! `/api/egress/allowed-domains` — the destinations the swarm lets agents
|
||||
//! reach through their hive's egress proxy.
|
||||
//! `/api/egress/allowed-domains` and `/api/egress/denied-domains` — the destinations the swarm lets
|
||||
//! agents reach through their hive's egress proxy, and the ones it refuses.
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
|
|
@ -33,6 +33,40 @@ pub struct AllowedDomains {
|
|||
pub entries: Vec<AllowedDomain>,
|
||||
}
|
||||
|
||||
/// One refused destination and who it is refused for.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[cfg_attr(feature = "schema", derive(utoipa::ToSchema))]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct DeniedDomain {
|
||||
/// A DNS name written as in [`AllowedDomain::domain`]: at least two
|
||||
/// labels, lower-cased, one leading dot for the name and its subdomains.
|
||||
#[cfg_attr(feature = "schema", schema(example = ".example.com"))]
|
||||
pub domain: String,
|
||||
/// The agent this entry is for. Absent means every agent in the swarm. An
|
||||
/// agent's entry adds to the swarm-wide ones and never lifts one.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
#[cfg_attr(feature = "schema", schema(example = "atlas"))]
|
||||
pub agent: Option<String>,
|
||||
}
|
||||
|
||||
/// The swarm's deny list: every entry, swarm-wide and per agent, sorted, and
|
||||
/// whether every hive has taken it.
|
||||
#[derive(Debug, Default, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[cfg_attr(feature = "schema", derive(utoipa::ToSchema))]
|
||||
pub struct DeniedDomains {
|
||||
pub entries: Vec<DeniedDomain>,
|
||||
/// The revision of the stored list, `0` when nobody has written one. A
|
||||
/// hive reports the revision it last rendered into its proxy.
|
||||
#[serde(default)]
|
||||
pub revision: u64,
|
||||
/// The hives whose proxy is not known to refuse these names yet: they have
|
||||
/// not reported the list's revision (a hive that predates the deny list
|
||||
/// never does, a new one takes a minute or so), or have not reported at
|
||||
/// all. Empty when the list is empty and no hive reports otherwise.
|
||||
#[serde(default)]
|
||||
pub behind: Vec<String>,
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
|
@ -60,4 +94,23 @@ mod tests {
|
|||
);
|
||||
assert!(serde_json::from_str::<AllowedDomain>(r#"{"domain":"a.io","hive":"x"}"#).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_denied_entry_has_no_derived_flag_and_refuses_unknown_fields() {
|
||||
let swarm_wide = DeniedDomain {
|
||||
domain: ".example.com".to_owned(),
|
||||
agent: None,
|
||||
};
|
||||
let wire = r#"{"domain":".example.com"}"#;
|
||||
assert_eq!(serde_json::to_string(&swarm_wide).unwrap(), wire);
|
||||
assert_eq!(
|
||||
serde_json::from_str::<DeniedDomain>(wire).unwrap(),
|
||||
swarm_wide
|
||||
);
|
||||
// A deny is never automatic, so the allow list's `derived` is not a field.
|
||||
assert!(
|
||||
serde_json::from_str::<DeniedDomain>(r#"{"domain":"a.io","derived":true}"#).is_err()
|
||||
);
|
||||
assert!(serde_json::from_str::<DeniedDomain>(r#"{"domain":"a.io","hive":"x"}"#).is_err());
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -68,7 +68,10 @@ The swarm's control plane. The swarm UI and `swarmctl` are its clients.
|
|||
(`GET`/`POST`/`DELETE /api/egress/allowed-domains`); every hive reads it and
|
||||
renders the proxy's allow-list files, which add to the names the hive's own
|
||||
configuration lists. It also keeps a `derived` entry per agent for the host
|
||||
of its stored backend environment's `ANTHROPIC_BASE_URL` →
|
||||
of its stored backend environment's `ANTHROPIC_BASE_URL`. The deny list is a second
|
||||
value of that bucket (`GET`/`POST`/`DELETE /api/egress/denied-domains`) with no
|
||||
derived entries; every hive renders it into an include the proxy checks before
|
||||
any allow, and reports the revision it rendered in its status snapshot →
|
||||
[`docs/networking/network.md`](../docs/networking/network.md#egress-proxy-squid).
|
||||
- **Agent instructions** — each agent's `AGENTS.md` as its harness last
|
||||
published it into the `agents-md` KV bucket
|
||||
|
|
|
|||
755
swarm-controller/src/egress_deny.rs
Normal file
755
swarm-controller/src/egress_deny.rs
Normal file
|
|
@ -0,0 +1,755 @@
|
|||
//! `/api/egress/denied-domains` — the destinations the swarm refuses agents
|
||||
//! through their hive's egress proxy, whatever else allows them.
|
||||
//!
|
||||
//! The list is one value under the `deny` key of the `egress-allow` bucket
|
||||
//! (`swarm_wire::egress_allow`), which this daemon alone writes, beside the
|
||||
//! allow-list's key. Every hive reads it and renders the entries for the agents
|
||||
//! it runs: a swarm-wide entry is refused for everyone, an entry with an agent
|
||||
//! for that agent alone. The proxy checks the list before any allow rule, so an
|
||||
//! entry beats the base list, the hive's own names, the allow-list, the
|
||||
//! unrestricted flag and `log` mode alike, and an agent's entry can only add to
|
||||
//! the swarm-wide ones.
|
||||
//!
|
||||
//! Only an operator's request writes an entry. There is no derived entry: the
|
||||
//! stored type has no `derived` field, and nothing in this daemon reads an
|
||||
//! agent's environment or a linked account into this list.
|
||||
//!
|
||||
//! Each name goes into the proxy's configuration, so it is checked with
|
||||
//! `swarm_wire::egress_allow::normalize_domain` here and again by the hive.
|
||||
//! The base list is not consulted: refusing a name the proxy otherwise always
|
||||
//! allows is the point of the list.
|
||||
//!
|
||||
//! A hive reports the revision of the list it rendered in its status snapshot
|
||||
//! (`swarm_wire::hive_status::EGRESS_DENY_APPLIED`); [`behind`] names the hives
|
||||
//! whose report does not match, which the reads show as `behind`.
|
||||
|
||||
use axum::Json;
|
||||
use axum::extract::State;
|
||||
use axum::http::StatusCode;
|
||||
use problem_details::ProblemDetails;
|
||||
use swarm_controller_api::egress::{DeniedDomain, DeniedDomains};
|
||||
use swarm_queue_client::egress_allow;
|
||||
use swarm_wire::egress_allow::{DENY_KEY, DenyEntry, DenyList, MAX_ENTRIES, normalize_domain};
|
||||
use swarm_wire::hive_status::egress_deny_applied;
|
||||
|
||||
use crate::routes::{AppState, error_problem, wanted_error_status};
|
||||
use crate::schedules::{live, queue, read_or_default, refused, write_revision};
|
||||
use crate::status::{Freshness, HiveStatus};
|
||||
|
||||
/// How many times a losing writer re-reads and re-applies, as in
|
||||
/// `egress_allow::MAX_ATTEMPTS`.
|
||||
const MAX_ATTEMPTS: usize = 5;
|
||||
|
||||
/// The entry to store for `req`, or why it is refused.
|
||||
pub(crate) fn checked(req: &DeniedDomain) -> Result<DenyEntry, String> {
|
||||
let domain = normalize_domain(&req.domain)?;
|
||||
let agent = req
|
||||
.agent
|
||||
.as_deref()
|
||||
.map(|a| {
|
||||
hive_types::Ident::parse(a)
|
||||
.map(hive_types::Ident::into_string)
|
||||
.map_err(|reason| format!("agent {a:?}: {reason}"))
|
||||
})
|
||||
.transpose()?;
|
||||
Ok(DenyEntry { domain, agent })
|
||||
}
|
||||
|
||||
/// The hives among `rows` whose proxy is not known to refuse the list at
|
||||
/// `revision` yet.
|
||||
///
|
||||
/// A hive that reports a revision is behind when it differs. A hive that
|
||||
/// reports none (it predates the field, has not read the list, or has never
|
||||
/// reported) is behind only while the list holds an entry: it cannot be
|
||||
/// enforcing a name it was never told, but an empty list asks nothing of it.
|
||||
/// A hive the roster does not list is left out.
|
||||
pub(crate) fn behind(rows: &[HiveStatus], revision: u64, list_is_empty: bool) -> Vec<String> {
|
||||
rows.iter()
|
||||
.filter(|row| row.freshness != Freshness::Unknown)
|
||||
.filter(
|
||||
|row| match row.snapshot.as_ref().and_then(egress_deny_applied) {
|
||||
Some(applied) => applied != revision,
|
||||
None => !list_is_empty,
|
||||
},
|
||||
)
|
||||
.map(|row| row.name.clone())
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// The list as the API answers it.
|
||||
fn view(list: DenyList, revision: u64, behind: Vec<String>) -> DeniedDomains {
|
||||
let mut entries = list.entries;
|
||||
entries.sort();
|
||||
DeniedDomains {
|
||||
entries: entries
|
||||
.into_iter()
|
||||
.map(|e| DeniedDomain {
|
||||
domain: e.domain,
|
||||
agent: e.agent,
|
||||
})
|
||||
.collect(),
|
||||
revision,
|
||||
behind,
|
||||
}
|
||||
}
|
||||
|
||||
/// Which hives are behind `list` at `revision`. A status read that fails names
|
||||
/// every hive, since nothing then confirms any.
|
||||
async fn behind_hives(state: &AppState, revision: u64, list: &DenyList) -> Vec<String> {
|
||||
let Some(reader) = state.status.as_ref() else {
|
||||
return Vec::new();
|
||||
};
|
||||
match reader
|
||||
.view(&state.hives, std::time::SystemTime::now())
|
||||
.await
|
||||
{
|
||||
Ok(rows) => behind(&rows, revision, list.entries.is_empty()),
|
||||
Err(e) => {
|
||||
tracing::warn!(error = %format!("{e:#}"), "egress deny list: reading the hives' status failed");
|
||||
state.hives.iter().map(|h| h.name.clone()).collect()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn queue_problem(e: &anyhow::Error) -> ProblemDetails {
|
||||
tracing::warn!(error = %format!("{e:#}"), "egress deny list: queue failure");
|
||||
error_problem(wanted_error_status(e), &format!("{e:#}"))
|
||||
}
|
||||
|
||||
async fn open_or_create(
|
||||
client: &async_nats::Client,
|
||||
) -> Result<async_nats::jetstream::kv::Store, ProblemDetails> {
|
||||
egress_allow::open_or_create(client)
|
||||
.await
|
||||
.map_err(|e| queue_problem(&anyhow::Error::new(e)))
|
||||
}
|
||||
|
||||
#[utoipa::path(
|
||||
get,
|
||||
path = "/api/egress/denied-domains",
|
||||
responses(
|
||||
(status = 200, description = "every entry, swarm-wide and per agent, sorted, the \
|
||||
stored revision, and the hives that have not reported rendering it; empty when \
|
||||
the swarm has not set a list",
|
||||
body = DeniedDomains),
|
||||
(status = 503, description = "no swarm queue is configured on this host, or it is \
|
||||
not connected (problem+json)", body = String),
|
||||
(status = 500, description = "the list could not be read (problem+json)",
|
||||
body = String),
|
||||
),
|
||||
tag = "egress"
|
||||
)]
|
||||
pub(crate) async fn get_denied_domains(
|
||||
State(state): State<AppState>,
|
||||
) -> Result<Json<DeniedDomains>, ProblemDetails> {
|
||||
let client = queue(&state).map_err(refused)?;
|
||||
let Some(store) = egress_allow::open(&client)
|
||||
.await
|
||||
.map_err(|e| queue_problem(&anyhow::Error::new(e)))?
|
||||
else {
|
||||
return Ok(Json(DeniedDomains::default()));
|
||||
};
|
||||
let (list, revision, _) = read_or_default::<DenyList>(&store, DENY_KEY)
|
||||
.await
|
||||
.map_err(|e| queue_problem(&e))?;
|
||||
let revision = revision.unwrap_or(0);
|
||||
let behind = behind_hives(&state, revision, &list).await;
|
||||
Ok(Json(view(list, revision, behind)))
|
||||
}
|
||||
|
||||
/// What changing a list came to.
|
||||
enum Changed {
|
||||
/// Nothing to write.
|
||||
Same(DenyList, u64),
|
||||
/// The list to write.
|
||||
New(DenyList),
|
||||
}
|
||||
|
||||
/// `list` with `entry` added, or why it can't take another entry.
|
||||
fn added(mut list: DenyList, entry: &DenyEntry, revision: u64) -> Result<Changed, String> {
|
||||
if list.entries.contains(entry) {
|
||||
return Ok(Changed::Same(list, revision));
|
||||
}
|
||||
if list.entries.len() >= MAX_ENTRIES {
|
||||
return Err(format!(
|
||||
"the list holds {MAX_ENTRIES} entries, which is the most it takes"
|
||||
));
|
||||
}
|
||||
list.entries.push(entry.clone());
|
||||
list.entries.sort();
|
||||
Ok(Changed::New(list))
|
||||
}
|
||||
|
||||
/// `list` without `entry`.
|
||||
fn removed(mut list: DenyList, entry: &DenyEntry, revision: u64) -> Changed {
|
||||
if !list.entries.contains(entry) {
|
||||
return Changed::Same(list, revision);
|
||||
}
|
||||
list.entries.retain(|e| e != entry);
|
||||
Changed::New(list)
|
||||
}
|
||||
|
||||
/// Re-read the list and write `change` of it, retrying a lost write, and answer
|
||||
/// the list as it stands after.
|
||||
async fn apply(
|
||||
state: &AppState,
|
||||
client: &async_nats::Client,
|
||||
what: &str,
|
||||
change: impl Fn(DenyList, u64) -> Result<Changed, String>,
|
||||
) -> Result<Json<DeniedDomains>, ProblemDetails> {
|
||||
let store = open_or_create(client).await?;
|
||||
for _ in 0..MAX_ATTEMPTS {
|
||||
let (list, revision, _) = read_or_default::<DenyList>(&store, DENY_KEY)
|
||||
.await
|
||||
.map_err(|e| queue_problem(&e))?;
|
||||
let list = match change(list, revision.unwrap_or(0)) {
|
||||
Ok(Changed::Same(list, revision)) => {
|
||||
let behind = behind_hives(state, revision, &list).await;
|
||||
return Ok(Json(view(list, revision, behind)));
|
||||
}
|
||||
Ok(Changed::New(list)) => list,
|
||||
Err(why) => return Err(error_problem(StatusCode::BAD_REQUEST, &why)),
|
||||
};
|
||||
if let Some(written) = write_revision(&store, DENY_KEY, &list, revision)
|
||||
.await
|
||||
.map_err(|e| queue_problem(&e))?
|
||||
{
|
||||
tracing::info!(domain = %what, revision = written, "changed the egress deny list");
|
||||
let behind = behind_hives(state, written, &list).await;
|
||||
return Ok(Json(view(list, written, behind)));
|
||||
}
|
||||
}
|
||||
Err(queue_problem(&anyhow::anyhow!(
|
||||
"gave up changing {what} after {MAX_ATTEMPTS} conflicting writes"
|
||||
)))
|
||||
}
|
||||
|
||||
/// Refuse one destination, for every agent or for one.
|
||||
///
|
||||
/// Adding an entry that is already there changes nothing. The entry is
|
||||
/// refused before any allow rule on every hive that has taken the list.
|
||||
#[utoipa::path(
|
||||
post,
|
||||
path = "/api/egress/denied-domains",
|
||||
request_body = DeniedDomain,
|
||||
responses(
|
||||
(status = 200, description = "the whole list after the add", body = DeniedDomains),
|
||||
(status = 400, description = "the domain is not a DNS name (an IP address, a \
|
||||
wildcard, a single label and a port are refused), the agent is not an \
|
||||
identifier or not declared on any hive, or the list is full (problem+json)",
|
||||
body = String),
|
||||
(status = 422, description = "an unknown field"),
|
||||
(status = 503, description = "no swarm queue is configured on this host, or it is \
|
||||
not connected (problem+json)", body = String),
|
||||
(status = 500, description = "the list could not be written (problem+json)",
|
||||
body = String),
|
||||
),
|
||||
tag = "egress"
|
||||
)]
|
||||
pub(crate) async fn add_denied_domain(
|
||||
State(state): State<AppState>,
|
||||
Json(req): Json<DeniedDomain>,
|
||||
) -> Result<Json<DeniedDomains>, ProblemDetails> {
|
||||
let entry = checked(&req).map_err(|why| error_problem(StatusCode::BAD_REQUEST, &why))?;
|
||||
let client = queue(&state).map_err(refused)?;
|
||||
if let Some(agent) = &entry.agent
|
||||
&& !live(&state).await?.contains(agent)
|
||||
{
|
||||
return Err(error_problem(
|
||||
StatusCode::BAD_REQUEST,
|
||||
&format!("no hive's wanted state declares an agent {agent:?} that is not destroyed"),
|
||||
));
|
||||
}
|
||||
apply(&state, &client, &entry.domain, |list, revision| {
|
||||
added(list, &entry, revision)
|
||||
})
|
||||
.await
|
||||
}
|
||||
|
||||
/// Stop refusing one destination for the scope it was added with.
|
||||
///
|
||||
/// Removing an entry that is not there changes nothing. A swarm-wide entry is
|
||||
/// removed only by naming it without an agent: removing an agent's entry for
|
||||
/// the same name leaves it refused for everyone.
|
||||
#[utoipa::path(
|
||||
delete,
|
||||
path = "/api/egress/denied-domains",
|
||||
request_body = DeniedDomain,
|
||||
responses(
|
||||
(status = 200, description = "the whole list after the removal", body = DeniedDomains),
|
||||
(status = 400, description = "the domain is not a DNS name, or the agent is not an \
|
||||
identifier (problem+json)", body = String),
|
||||
(status = 422, description = "an unknown field"),
|
||||
(status = 503, description = "no swarm queue is configured on this host, or it is \
|
||||
not connected (problem+json)", body = String),
|
||||
(status = 500, description = "the list could not be written (problem+json)",
|
||||
body = String),
|
||||
),
|
||||
tag = "egress"
|
||||
)]
|
||||
pub(crate) async fn remove_denied_domain(
|
||||
State(state): State<AppState>,
|
||||
Json(req): Json<DeniedDomain>,
|
||||
) -> Result<Json<DeniedDomains>, ProblemDetails> {
|
||||
let entry = checked(&req).map_err(|why| error_problem(StatusCode::BAD_REQUEST, &why))?;
|
||||
let client = queue(&state).map_err(refused)?;
|
||||
apply(&state, &client, &entry.domain, |list, revision| {
|
||||
Ok(removed(list, &entry, revision))
|
||||
})
|
||||
.await
|
||||
}
|
||||
|
||||
/// Drop every entry of a destroyed agent: a name is free again once its agent
|
||||
/// is destroyed, and the next agent of that name would inherit what it left. A
|
||||
/// list that was never written stays unwritten. A failure is logged and does
|
||||
/// not stop the destroy, as in `egress_allow::clear_on_destroy`.
|
||||
pub(crate) async fn clear_on_destroy(state: &AppState, hive: &str, agent: &str) {
|
||||
let cleared = async {
|
||||
let client = queue(state).map_err(|(_, why)| anyhow::anyhow!(why))?;
|
||||
let Some(store) = egress_allow::open(&client).await? else {
|
||||
return Ok(());
|
||||
};
|
||||
for _ in 0..MAX_ATTEMPTS {
|
||||
let (mut list, revision, _) = read_or_default::<DenyList>(&store, DENY_KEY).await?;
|
||||
let before = list.entries.len();
|
||||
list.entries.retain(|e| e.agent.as_deref() != Some(agent));
|
||||
if list.entries.len() == before {
|
||||
return Ok(());
|
||||
}
|
||||
if write_revision(&store, DENY_KEY, &list, revision)
|
||||
.await?
|
||||
.is_some()
|
||||
{
|
||||
return Ok(());
|
||||
}
|
||||
}
|
||||
anyhow::bail!("gave up after {MAX_ATTEMPTS} conflicting writes")
|
||||
}
|
||||
.await;
|
||||
match cleared {
|
||||
Ok(()) => tracing::info!(hive, agent, "destroyed agent's denied domains cleared"),
|
||||
Err(e) => tracing::error!(
|
||||
hive,
|
||||
agent,
|
||||
error = %format!("{e:#}"),
|
||||
"clearing this agent's denied domains FAILED; it was destroyed anyway, so a new \
|
||||
agent of the same name inherits them. Declaring the agent destroyed again re-runs this."
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::agent_limits::tests::{declared, state};
|
||||
use crate::test_nats::TestNats;
|
||||
use swarm_wire::hive_status::EGRESS_DENY_APPLIED;
|
||||
|
||||
fn entry(domain: &str, agent: Option<&str>) -> DeniedDomain {
|
||||
DeniedDomain {
|
||||
domain: domain.to_owned(),
|
||||
agent: agent.map(str::to_owned),
|
||||
}
|
||||
}
|
||||
|
||||
async fn add(
|
||||
state: &AppState,
|
||||
domain: &str,
|
||||
agent: Option<&str>,
|
||||
) -> Result<DeniedDomains, ProblemDetails> {
|
||||
add_denied_domain(State(state.clone()), Json(entry(domain, agent)))
|
||||
.await
|
||||
.map(|j| j.0)
|
||||
}
|
||||
|
||||
async fn remove(
|
||||
state: &AppState,
|
||||
domain: &str,
|
||||
agent: Option<&str>,
|
||||
) -> Result<DeniedDomains, ProblemDetails> {
|
||||
remove_denied_domain(State(state.clone()), Json(entry(domain, agent)))
|
||||
.await
|
||||
.map(|j| j.0)
|
||||
}
|
||||
|
||||
async fn read(state: &AppState) -> DeniedDomains {
|
||||
get_denied_domains(State(state.clone()))
|
||||
.await
|
||||
.expect("the list is read")
|
||||
.0
|
||||
}
|
||||
|
||||
async fn listed(state: &AppState) -> Vec<(String, Option<String>)> {
|
||||
read(state)
|
||||
.await
|
||||
.entries
|
||||
.into_iter()
|
||||
.map(|e| (e.domain, e.agent))
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// The raw value a hive reads under `key`, or `None` for no key.
|
||||
async fn published(client: &async_nats::Client, key: &str) -> Option<String> {
|
||||
egress_allow::open_or_create(client)
|
||||
.await
|
||||
.expect("bucket")
|
||||
.get(key)
|
||||
.await
|
||||
.expect("read")
|
||||
.map(|v| String::from_utf8(v.to_vec()).expect("utf-8"))
|
||||
}
|
||||
|
||||
fn row(name: &str, freshness: Freshness, applied: Option<u64>) -> HiveStatus {
|
||||
HiveStatus {
|
||||
name: name.to_owned(),
|
||||
domain: None,
|
||||
dashboard: None,
|
||||
freshness,
|
||||
last_seen_unix: None,
|
||||
age_seconds: None,
|
||||
snapshot: applied.map(|r| serde_json::json!({ EGRESS_DENY_APPLIED: r })),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_name_is_stored_normalised_and_an_ip_literal_is_refused() {
|
||||
let stored = checked(&entry(" Evil.COM ", Some("atlas"))).expect("a name");
|
||||
assert_eq!(stored.domain, "evil.com");
|
||||
assert_eq!(stored.agent.as_deref(), Some("atlas"));
|
||||
let dotted = checked(&entry(".evil.com", None)).expect("dot form");
|
||||
assert_eq!(dotted.domain, ".evil.com");
|
||||
|
||||
for bad in [
|
||||
"1.2.3.4",
|
||||
"[::1]",
|
||||
"*.example.com",
|
||||
"localhost",
|
||||
"example.com:443",
|
||||
"example.com\nhttp_access allow all",
|
||||
] {
|
||||
assert!(checked(&entry(bad, None)).is_err(), "{bad:?} was accepted");
|
||||
}
|
||||
for agent in ["", "At Las", "a/b", "../argus"] {
|
||||
assert!(
|
||||
checked(&entry("evil.com", Some(agent))).is_err(),
|
||||
"agent {agent:?} was accepted"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// The base list is no reason to refuse a deny: denying a name that is
|
||||
/// otherwise always allowed is what the list is for.
|
||||
#[tokio::test]
|
||||
async fn a_name_the_base_list_allows_can_be_denied() {
|
||||
let nats = TestNats::start();
|
||||
let client = nats.client().await;
|
||||
let state = state(&client);
|
||||
// Control: the allow list's own check refuses this name.
|
||||
assert!(
|
||||
crate::egress_allow::checked(
|
||||
&swarm_controller_api::egress::AllowedDomain {
|
||||
domain: "api.anthropic.com".to_owned(),
|
||||
agent: None,
|
||||
derived: false,
|
||||
},
|
||||
&["api.anthropic.com".to_owned()],
|
||||
)
|
||||
.is_err()
|
||||
);
|
||||
add(&state, "api.anthropic.com", None)
|
||||
.await
|
||||
.expect("denied despite the base list");
|
||||
}
|
||||
|
||||
/// Through the real handlers and a real `nats-server`: an entry lands under
|
||||
/// the deny key a hive reads and never under the allow key, the two scopes
|
||||
/// of one name are two entries, a repeat writes nothing, and removing an
|
||||
/// agent's entry leaves the swarm-wide one in force.
|
||||
///
|
||||
/// The published bytes are pinned literally because hive-c0re's
|
||||
/// `workers::squid_deny` tests decode this same string.
|
||||
#[tokio::test]
|
||||
async fn the_route_writes_the_list_a_hive_reads_and_a_scoped_removal_never_lifts_the_swarm_wide_entry()
|
||||
{
|
||||
let nats = TestNats::start();
|
||||
let client = nats.client().await;
|
||||
let state = declared(&client).await;
|
||||
|
||||
add(&state, "evil.com", None).await.expect("swarm-wide");
|
||||
add(&state, "bad.io", Some("atlas")).await.expect("atlas's");
|
||||
add(&state, "evil.com", Some("atlas"))
|
||||
.await
|
||||
.expect("the same name for atlas alone");
|
||||
let before = published(&client, DENY_KEY).await;
|
||||
add(&state, "EVIL.com", None).await.expect("a repeat");
|
||||
assert_eq!(
|
||||
published(&client, DENY_KEY).await,
|
||||
before,
|
||||
"a repeat writes nothing"
|
||||
);
|
||||
|
||||
assert_eq!(
|
||||
published(&client, DENY_KEY).await.as_deref(),
|
||||
Some(
|
||||
r#"{"entries":[{"domain":"bad.io","agent":"atlas"},{"domain":"evil.com"},{"domain":"evil.com","agent":"atlas"}]}"#
|
||||
)
|
||||
);
|
||||
assert_eq!(
|
||||
published(&client, swarm_wire::egress_allow::KEY).await,
|
||||
None,
|
||||
"a deny is not an allow entry"
|
||||
);
|
||||
|
||||
remove(&state, "evil.com", Some("atlas"))
|
||||
.await
|
||||
.expect("removes atlas's");
|
||||
assert_eq!(
|
||||
listed(&state).await,
|
||||
[
|
||||
("bad.io".to_owned(), Some("atlas".to_owned())),
|
||||
("evil.com".to_owned(), None),
|
||||
],
|
||||
"the swarm-wide deny stays after atlas's is removed"
|
||||
);
|
||||
let before = published(&client, DENY_KEY).await;
|
||||
remove(&state, "evil.com", Some("atlas"))
|
||||
.await
|
||||
.expect("already gone");
|
||||
assert_eq!(published(&client, DENY_KEY).await, before);
|
||||
|
||||
remove(&state, "evil.com", None).await.expect("swarm-wide");
|
||||
remove(&state, "bad.io", Some("atlas"))
|
||||
.await
|
||||
.expect("atlas's");
|
||||
assert_eq!(
|
||||
published(&client, DENY_KEY).await.as_deref(),
|
||||
Some(r#"{"entries":[]}"#),
|
||||
"the last removal leaves an empty list"
|
||||
);
|
||||
}
|
||||
|
||||
/// An allow-list write leaves the deny list alone and the other way round.
|
||||
#[tokio::test]
|
||||
async fn the_allow_list_and_the_deny_list_are_separate_values() {
|
||||
let nats = TestNats::start();
|
||||
let client = nats.client().await;
|
||||
let state = declared(&client).await;
|
||||
|
||||
add(&state, "evil.com", None).await.expect("a deny");
|
||||
let allowed = crate::egress_allow::add_allowed_domain(
|
||||
State(state.clone()),
|
||||
Json(swarm_controller_api::egress::AllowedDomain {
|
||||
domain: "github.com".to_owned(),
|
||||
agent: None,
|
||||
derived: false,
|
||||
}),
|
||||
)
|
||||
.await
|
||||
.expect("an allow");
|
||||
assert_eq!(allowed.0.entries.len(), 1);
|
||||
assert_eq!(
|
||||
published(&client, DENY_KEY).await.as_deref(),
|
||||
Some(r#"{"entries":[{"domain":"evil.com"}]}"#)
|
||||
);
|
||||
assert_eq!(
|
||||
published(&client, swarm_wire::egress_allow::KEY)
|
||||
.await
|
||||
.as_deref(),
|
||||
Some(r#"{"entries":[{"domain":"github.com"}]}"#)
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_read_before_any_entry_answers_empty_and_creates_no_bucket() {
|
||||
let nats = TestNats::start();
|
||||
let client = nats.client().await;
|
||||
let state = state(&client);
|
||||
|
||||
let read = read(&state).await;
|
||||
assert!(read.entries.is_empty() && read.revision == 0 && read.behind.is_empty());
|
||||
assert!(
|
||||
egress_allow::open(&client)
|
||||
.await
|
||||
.expect("the queue answers")
|
||||
.is_none()
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn an_agent_no_hive_declares_is_refused_and_nothing_is_written() {
|
||||
let nats = TestNats::start();
|
||||
let client = nats.client().await;
|
||||
let state = declared(&client).await;
|
||||
|
||||
let problem = add(&state, "evil.com", Some("iris"))
|
||||
.await
|
||||
.expect_err("no hive declares iris");
|
||||
assert_eq!(problem.status, Some(StatusCode::BAD_REQUEST), "{problem:?}");
|
||||
assert_eq!(published(&client, DENY_KEY).await, None);
|
||||
add(&state, "evil.com", Some("atlas"))
|
||||
.await
|
||||
.expect("control: a declared agent is accepted");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_full_list_takes_no_new_entry_but_a_present_one_is_not_a_new_one() {
|
||||
let full = DenyList {
|
||||
entries: (0..MAX_ENTRIES)
|
||||
.map(|i| DenyEntry {
|
||||
domain: format!("host{i}.example.com"),
|
||||
agent: None,
|
||||
})
|
||||
.collect(),
|
||||
};
|
||||
let new = DenyEntry {
|
||||
domain: "evil.com".to_owned(),
|
||||
agent: None,
|
||||
};
|
||||
let why = added(full.clone(), &new, 1).err().expect("a full list");
|
||||
assert!(why.contains("1000"), "{why}");
|
||||
let present = full.entries[0].clone();
|
||||
assert!(matches!(
|
||||
added(full.clone(), &present, 1),
|
||||
Ok(Changed::Same(..))
|
||||
));
|
||||
let mut room = full;
|
||||
room.entries.pop();
|
||||
let Ok(Changed::New(list)) = added(room, &new, 1) else {
|
||||
panic!("a list with room takes the entry");
|
||||
};
|
||||
assert_eq!(list.entries.len(), MAX_ENTRIES);
|
||||
}
|
||||
|
||||
/// A hive is behind unless it reports the list's revision; one that reports
|
||||
/// nothing is behind only while the list asks something of it.
|
||||
#[test]
|
||||
fn a_hive_is_behind_unless_it_reports_the_revision_it_should_have() {
|
||||
let rows = [
|
||||
row("current", Freshness::Fresh, Some(7)),
|
||||
row("older", Freshness::Fresh, Some(5)),
|
||||
row("silent-old", Freshness::Fresh, None),
|
||||
row("never", Freshness::NeverReported, None),
|
||||
row("stale-current", Freshness::Stale, Some(7)),
|
||||
row("stranger", Freshness::Unknown, None),
|
||||
];
|
||||
assert_eq!(behind(&rows, 7, false), ["older", "silent-old", "never"]);
|
||||
// An empty list asks nothing of a hive that reports nothing, but a hive
|
||||
// that reports a different revision still holds names the list dropped.
|
||||
assert_eq!(behind(&rows, 7, true), ["older"]);
|
||||
assert_eq!(
|
||||
behind(&rows, 0, true),
|
||||
["current", "older", "stale-current"]
|
||||
);
|
||||
}
|
||||
|
||||
/// Through the status bucket: after an add every hive is behind until it
|
||||
/// publishes the revision the add answered.
|
||||
#[tokio::test]
|
||||
async fn a_hive_leaves_behind_when_its_snapshot_carries_the_revision() {
|
||||
let nats = TestNats::start();
|
||||
let client = nats.client().await;
|
||||
let state = declared(&client).await;
|
||||
|
||||
let added = add(&state, "evil.com", None).await.expect("a deny");
|
||||
assert!(added.revision > 0);
|
||||
assert_eq!(added.behind, ["pr1ma"], "it has not reported");
|
||||
|
||||
let status = swarm_queue_client::status::open_or_create(&client)
|
||||
.await
|
||||
.expect("status bucket");
|
||||
let publish = |applied: u64| {
|
||||
let body = serde_json::json!({
|
||||
"status": "ok", "warnings": [], EGRESS_DENY_APPLIED: applied
|
||||
});
|
||||
let status = status.clone();
|
||||
async move {
|
||||
status
|
||||
.put("pr1ma", body.to_string().into())
|
||||
.await
|
||||
.expect("publish");
|
||||
}
|
||||
};
|
||||
publish(added.revision - 1).await;
|
||||
assert_eq!(read(&state).await.behind, ["pr1ma"], "an older revision");
|
||||
publish(added.revision).await;
|
||||
let now = read(&state).await;
|
||||
assert_eq!(now.revision, added.revision);
|
||||
assert!(now.behind.is_empty(), "{:?}", now.behind);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn destroying_an_agent_drops_its_denies_and_keeps_the_rest() {
|
||||
let nats = TestNats::start();
|
||||
let client = nats.client().await;
|
||||
let state = declared(&client).await;
|
||||
add(&state, "evil.com", None).await.expect("swarm-wide");
|
||||
add(&state, "bad.io", Some("atlas")).await.expect("atlas's");
|
||||
|
||||
clear_on_destroy(&state, "pr1ma", "atlas").await;
|
||||
assert_eq!(listed(&state).await, [("evil.com".to_owned(), None)]);
|
||||
}
|
||||
|
||||
async fn call(app: &axum::Router, method: &str, body: &'static str, path: &str) -> StatusCode {
|
||||
use tower::ServiceExt as _;
|
||||
app.clone()
|
||||
.oneshot(
|
||||
axum::http::Request::builder()
|
||||
.method(method)
|
||||
.uri(path)
|
||||
.header(axum::http::header::CONTENT_TYPE, "application/json")
|
||||
.body(axum::body::Body::from(body))
|
||||
.expect("request"),
|
||||
)
|
||||
.await
|
||||
.expect("router answers")
|
||||
.status()
|
||||
}
|
||||
|
||||
/// Through the real router: an IP literal or a wildcard is a 400 and writes
|
||||
/// nothing, an unknown field (a `derived` flag included) a 422, and the
|
||||
/// well-formed body after them is the control that the route reaches the
|
||||
/// deny key.
|
||||
#[tokio::test]
|
||||
async fn the_router_refuses_bad_names_and_a_derived_flag_before_the_queue() {
|
||||
let nats = TestNats::start();
|
||||
let client = nats.client().await;
|
||||
let app = crate::startup::build_app(declared(&client).await);
|
||||
let path = "/api/egress/denied-domains";
|
||||
|
||||
for body in [
|
||||
r#"{"domain":"1.2.3.4"}"#,
|
||||
r#"{"domain":"*.example.com"}"#,
|
||||
r#"{"domain":""}"#,
|
||||
] {
|
||||
assert_eq!(
|
||||
call(&app, "POST", body, path).await,
|
||||
StatusCode::BAD_REQUEST,
|
||||
"{body}"
|
||||
);
|
||||
}
|
||||
for body in [
|
||||
r#"{"domain":"evil.com","derived":true}"#,
|
||||
r#"{"domain":"evil.com","hive":"pr1ma"}"#,
|
||||
] {
|
||||
assert_eq!(
|
||||
call(&app, "POST", body, path).await,
|
||||
StatusCode::UNPROCESSABLE_ENTITY,
|
||||
"{body}"
|
||||
);
|
||||
}
|
||||
assert_eq!(published(&client, DENY_KEY).await, None);
|
||||
|
||||
assert_eq!(
|
||||
call(&app, "POST", r#"{"domain":"evil.com"}"#, path).await,
|
||||
StatusCode::OK
|
||||
);
|
||||
assert_eq!(
|
||||
published(&client, DENY_KEY).await.as_deref(),
|
||||
Some(r#"{"entries":[{"domain":"evil.com"}]}"#)
|
||||
);
|
||||
assert_eq!(
|
||||
call(&app, "DELETE", r#"{"domain":"evil.com"}"#, path).await,
|
||||
StatusCode::OK
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
@ -49,6 +49,7 @@ mod claude_login;
|
|||
mod config_drift;
|
||||
mod config_pr;
|
||||
mod egress_allow;
|
||||
mod egress_deny;
|
||||
mod forge;
|
||||
mod forge_account;
|
||||
mod forge_events;
|
||||
|
|
|
|||
|
|
@ -19,7 +19,7 @@ use crate::routes::hives::HiveEntry;
|
|||
use crate::routes::{AppState, StatusUnavailable, error_problem, wanted_error_status};
|
||||
use crate::{
|
||||
agent_archetype, agent_identity, agent_limits, agent_permissions, agent_status,
|
||||
agents_md_initial, auth, egress_allow, inference, schedules, wanted,
|
||||
agents_md_initial, auth, egress_allow, egress_deny, inference, schedules, wanted,
|
||||
};
|
||||
|
||||
/// The swarm's agent roster — every agent the swarm holds an identity for.
|
||||
|
|
@ -217,6 +217,7 @@ pub(crate) async fn set_agent_state(
|
|||
agent_archetype::clear_on_destroy(&state, &hive, &agent).await;
|
||||
agents_md_initial::clear_on_destroy(&state, &hive, &agent).await;
|
||||
egress_allow::clear_on_destroy(&state, &hive, &agent).await;
|
||||
egress_deny::clear_on_destroy(&state, &hive, &agent).await;
|
||||
}
|
||||
Ok(Json(render(&declaration)))
|
||||
}
|
||||
|
|
|
|||
|
|
@ -31,9 +31,10 @@ use crate::{
|
|||
agent_control, agent_custom, agent_icon, agent_input, agent_limits, agent_login,
|
||||
agent_permissions, agent_renewal, agent_state_stream, agent_status, agents_md, alerts,
|
||||
archetypes, auth, backend_env, builder, claude_login, config_drift, config_pr, egress_allow,
|
||||
forge, forge_account, github_account, gitlab_account, inference, issue_report, linked_accounts,
|
||||
matrix_account, message_stream, notify_flows, notify_router_api, pulls, queue_identity,
|
||||
review_classifier, schedules, status, subagent_term, term_stream, wanted, webhook,
|
||||
egress_deny, forge, forge_account, github_account, gitlab_account, inference, issue_report,
|
||||
linked_accounts, matrix_account, message_stream, notify_flows, notify_router_api, pulls,
|
||||
queue_identity, review_classifier, schedules, status, subagent_term, term_stream, wanted,
|
||||
webhook,
|
||||
};
|
||||
|
||||
/// Where the daemon binds, overridable via `SWARM_CONTROLLER_SOCKET`.
|
||||
|
|
@ -649,6 +650,11 @@ pub(crate) fn build_app(state: AppState) -> axum::Router {
|
|||
egress_allow::add_allowed_domain,
|
||||
egress_allow::remove_allowed_domain
|
||||
))
|
||||
.routes(routes!(
|
||||
egress_deny::get_denied_domains,
|
||||
egress_deny::add_denied_domain,
|
||||
egress_deny::remove_denied_domain
|
||||
))
|
||||
.routes(routes!(archetypes::get_archetypes))
|
||||
.routes(routes!(config_prs::get_config_prs))
|
||||
.routes(routes!(agents::create_agent))
|
||||
|
|
|
|||
|
|
@ -613,6 +613,9 @@ impl Policy {
|
|||
// The egress allow-list (`swarm_wire::egress_allow`): one key, read with
|
||||
// one direct get, so exactly that subject and no other key of the bucket.
|
||||
subjects.push(swarm_wire::egress_allow::direct_get_subject());
|
||||
// The deny list is a second key of that bucket, read the same way; the
|
||||
// allow-list grant does not cover it, since it names one key.
|
||||
subjects.push(swarm_wire::egress_allow::deny_direct_get_subject());
|
||||
// The archetype an agent was created from
|
||||
// (`swarm_queue_client::agent_archetype`), read the same way and every
|
||||
// key for the same reason.
|
||||
|
|
@ -984,6 +987,48 @@ mod tests {
|
|||
}
|
||||
}
|
||||
|
||||
/// A hive reads the deny list with its own grant, not the allow-list's, and
|
||||
/// can change none of it. An agent's token reaches neither key, so nothing an
|
||||
/// agent holds can write the list a hive enforces.
|
||||
#[test]
|
||||
fn a_hive_reads_the_deny_list_and_no_agent_reaches_it() {
|
||||
use swarm_wire::egress_allow::{
|
||||
STREAM, deny_direct_get_subject, deny_key_subject, direct_get_subject, key_subject,
|
||||
};
|
||||
|
||||
let p = policy()
|
||||
.permissions("hive-alpha")
|
||||
.expect("a hive is admitted");
|
||||
let may = |subject: &str| p.publish.iter().any(|g| covers(g, subject));
|
||||
assert!(may(&deny_direct_get_subject()), "cannot read the deny list");
|
||||
for forbidden in [
|
||||
deny_key_subject(),
|
||||
format!("$JS.API.STREAM.PURGE.{STREAM}"),
|
||||
format!("$JS.API.STREAM.MSG.DELETE.{STREAM}"),
|
||||
] {
|
||||
assert!(!may(&forbidden), "hive alpha may publish {forbidden}");
|
||||
}
|
||||
|
||||
let agent = policy()
|
||||
.with_agent_token_subjects(vec!["$SWARM.term.{agent}".to_owned()])
|
||||
.expect("a per-agent template is valid")
|
||||
.agent_token_permissions("atlas")
|
||||
.expect("configured");
|
||||
let agent_may = |subject: &str| agent.publish.iter().any(|g| covers(g, subject));
|
||||
for reached in [
|
||||
deny_key_subject(),
|
||||
deny_direct_get_subject(),
|
||||
key_subject(),
|
||||
direct_get_subject(),
|
||||
] {
|
||||
assert!(!agent_may(&reached), "an agent may publish {reached}");
|
||||
}
|
||||
// Control: the lookup finds a subject the agent is granted.
|
||||
assert!(agent_may(
|
||||
&swarm_queue_client::operator_profiles::direct_get_subject()
|
||||
));
|
||||
}
|
||||
|
||||
/// A hive reads any agent's archetype and can change none of them, and the
|
||||
/// grant reaches no other bucket.
|
||||
#[test]
|
||||
|
|
|
|||
|
|
@ -56,7 +56,22 @@ pub enum Read {
|
|||
NotSet,
|
||||
}
|
||||
|
||||
/// Read the current list with one direct get.
|
||||
/// What reading the deny list found.
|
||||
#[cfg(feature = "egress-allow")]
|
||||
#[derive(Debug)]
|
||||
pub enum DenyRead {
|
||||
/// The controller has written a deny list, which may be empty, at this
|
||||
/// revision of its key: the stream sequence of the write, which is what the
|
||||
/// controller's own read of the key reports as its revision.
|
||||
Current {
|
||||
list: swarm_wire::egress_allow::DenyList,
|
||||
revision: u64,
|
||||
},
|
||||
/// There is no deny list to read, for the reasons [`Read::NotSet`] gives.
|
||||
NotSet,
|
||||
}
|
||||
|
||||
/// Read the current allow-list with one direct get.
|
||||
///
|
||||
/// # Errors
|
||||
///
|
||||
|
|
@ -66,15 +81,50 @@ pub enum Read {
|
|||
pub async fn read(
|
||||
client: &async_nats::Client,
|
||||
) -> Result<Read, Box<dyn std::error::Error + Send + Sync>> {
|
||||
let Some(message) = last_value(client, &swarm_wire::egress_allow::key_subject()).await? else {
|
||||
return Ok(Read::NotSet);
|
||||
};
|
||||
Ok(Read::Current(serde_json::from_slice(&message.payload)?))
|
||||
}
|
||||
|
||||
/// Read the current deny list with one direct get.
|
||||
///
|
||||
/// # Errors
|
||||
///
|
||||
/// As [`read`].
|
||||
#[cfg(feature = "egress-allow")]
|
||||
pub async fn read_deny(
|
||||
client: &async_nats::Client,
|
||||
) -> Result<DenyRead, Box<dyn std::error::Error + Send + Sync>> {
|
||||
let Some(message) = last_value(client, &swarm_wire::egress_allow::deny_key_subject()).await?
|
||||
else {
|
||||
return Ok(DenyRead::NotSet);
|
||||
};
|
||||
Ok(DenyRead::Current {
|
||||
list: serde_json::from_slice(&message.payload)?,
|
||||
revision: message.sequence,
|
||||
})
|
||||
}
|
||||
|
||||
/// The last message under `subject` of the bucket's stream, or `None` when
|
||||
/// there is none or it is a KV delete marker.
|
||||
#[cfg(feature = "egress-allow")]
|
||||
async fn last_value(
|
||||
client: &async_nats::Client,
|
||||
subject: &str,
|
||||
) -> Result<
|
||||
Option<async_nats::jetstream::message::StreamMessage>,
|
||||
Box<dyn std::error::Error + Send + Sync>,
|
||||
> {
|
||||
use async_nats::jetstream::stream::DirectGetErrorKind;
|
||||
use swarm_wire::egress_allow::{STREAM, key_subject};
|
||||
use swarm_wire::egress_allow::STREAM;
|
||||
|
||||
let js = async_nats::jetstream::new(client.clone());
|
||||
let stream = js.get_stream_no_info(STREAM).await?;
|
||||
let message = match stream.direct_get_last_for_subject(key_subject()).await {
|
||||
let message = match stream.direct_get_last_for_subject(subject).await {
|
||||
Ok(message) => message,
|
||||
Err(e) if matches!(e.kind(), DirectGetErrorKind::NotFound) => return Ok(Read::NotSet),
|
||||
Err(e) if is_no_responders(&e) => return Ok(Read::NotSet),
|
||||
Err(e) if matches!(e.kind(), DirectGetErrorKind::NotFound) => return Ok(None),
|
||||
Err(e) if is_no_responders(&e) => return Ok(None),
|
||||
Err(e) => return Err(e.into()),
|
||||
};
|
||||
// A KV delete or purge is stored as an empty marker carrying this header.
|
||||
|
|
@ -82,10 +132,7 @@ pub async fn read(
|
|||
.headers
|
||||
.get("KV-Operation")
|
||||
.is_some_and(|op| matches!(op.as_str(), "DEL" | "PURGE"));
|
||||
if removed {
|
||||
return Ok(Read::NotSet);
|
||||
}
|
||||
Ok(Read::Current(serde_json::from_slice(&message.payload)?))
|
||||
Ok((!removed).then_some(message))
|
||||
}
|
||||
|
||||
#[cfg(feature = "egress-allow")]
|
||||
|
|
@ -96,3 +143,75 @@ fn is_no_responders(e: &async_nats::jetstream::stream::DirectGetError) -> bool {
|
|||
if status == async_nats::StatusCode::NO_RESPONDERS
|
||||
)
|
||||
}
|
||||
|
||||
#[cfg(all(test, feature = "kv", feature = "egress-allow"))]
|
||||
mod tests {
|
||||
use swarm_wire::egress_allow::{DENY_KEY, DenyEntry, DenyList, KEY};
|
||||
|
||||
use super::{DenyRead, Read, open_or_create, read, read_deny};
|
||||
use crate::test_nats::TestNats;
|
||||
|
||||
/// The deny list is read from its own key with the revision of the write,
|
||||
/// the allow-list's key is not mistaken for it, and a deleted or never
|
||||
/// written key reads as unset.
|
||||
#[tokio::test]
|
||||
async fn the_deny_list_is_read_from_its_own_key_with_the_revision_it_was_written_at() {
|
||||
let nats = TestNats::start();
|
||||
let client = nats.client().await;
|
||||
|
||||
assert!(matches!(
|
||||
read_deny(&client).await.expect("no bucket"),
|
||||
DenyRead::NotSet
|
||||
));
|
||||
let store = open_or_create(&client).await.expect("bucket");
|
||||
assert!(matches!(
|
||||
read_deny(&client).await.expect("no key"),
|
||||
DenyRead::NotSet
|
||||
));
|
||||
|
||||
let list = DenyList {
|
||||
entries: vec![DenyEntry {
|
||||
domain: ".evil.example".to_owned(),
|
||||
agent: None,
|
||||
}],
|
||||
};
|
||||
store
|
||||
.put(
|
||||
KEY,
|
||||
br#"{"entries":[{"domain":"allowed.example"}]}"#.to_vec().into(),
|
||||
)
|
||||
.await
|
||||
.expect("an allow-list write");
|
||||
assert!(
|
||||
matches!(
|
||||
read_deny(&client).await.expect("still unset"),
|
||||
DenyRead::NotSet
|
||||
),
|
||||
"the allow-list's key is not the deny list"
|
||||
);
|
||||
let revision = store
|
||||
.put(DENY_KEY, serde_json::to_vec(&list).expect("encodes").into())
|
||||
.await
|
||||
.expect("a deny-list write");
|
||||
match read_deny(&client).await.expect("read") {
|
||||
DenyRead::Current {
|
||||
list: got,
|
||||
revision: at,
|
||||
} => {
|
||||
assert_eq!(got, list);
|
||||
assert_eq!(at, revision, "the revision the controller's write answered");
|
||||
}
|
||||
DenyRead::NotSet => panic!("the list was written"),
|
||||
}
|
||||
assert!(
|
||||
matches!(read(&client).await.expect("allow read"), Read::Current(l) if l.entries.len() == 1),
|
||||
"control: the allow-list read still finds its own key"
|
||||
);
|
||||
|
||||
store.delete(DENY_KEY).await.expect("deleted");
|
||||
assert!(matches!(
|
||||
read_deny(&client).await.expect("deleted"),
|
||||
DenyRead::NotSet
|
||||
));
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -487,7 +487,14 @@ pub mod forge_events;
|
|||
#[cfg(feature = "router-todo")]
|
||||
pub mod router_todo;
|
||||
|
||||
#[cfg(all(test, any(feature = "forge-events", feature = "router-todo")))]
|
||||
#[cfg(all(
|
||||
test,
|
||||
any(
|
||||
feature = "forge-events",
|
||||
feature = "router-todo",
|
||||
all(feature = "kv", feature = "egress-allow")
|
||||
)
|
||||
))]
|
||||
mod test_nats;
|
||||
|
||||
/// Only the fields this needs; authelia returns several.
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
//! The egress allow-list KV bucket: one key holding the destination names the
|
||||
//! swarm lets agents reach through their hive's squid.
|
||||
//! swarm lets agents reach through their hive's squid, and a second holding the
|
||||
//! names it refuses.
|
||||
//!
|
||||
//! **swarm-controller is the only writer**, and creates the bucket on its
|
||||
//! first write. Each [`Entry`] is a domain with a scope: every agent in the
|
||||
|
|
@ -10,6 +11,11 @@
|
|||
//! covers the bucket read-only, and renders the entries for the agents it runs.
|
||||
//! The names go into squid's configuration, so [`normalize_domain`] is the one
|
||||
//! gate both ends use: the controller at write, the hive again at render.
|
||||
//!
|
||||
//! The deny list is its own key ([`DENY_KEY`]) of the same bucket, with its own
|
||||
//! type ([`DenyList`]): a reader that predates it never reads the key, and a
|
||||
//! controller that predates it never rewrites it, so neither side can drop the
|
||||
//! other's entries by writing its own value back.
|
||||
|
||||
/// The KV bucket the list is published into.
|
||||
pub const BUCKET: &str = "egress-allow";
|
||||
|
|
@ -32,6 +38,21 @@ pub fn direct_get_subject() -> String {
|
|||
format!("$JS.API.DIRECT.GET.{STREAM}.{}", key_subject())
|
||||
}
|
||||
|
||||
/// The key the deny list is stored under, beside [`KEY`].
|
||||
pub const DENY_KEY: &str = "deny";
|
||||
|
||||
/// The subject the deny list is stored under.
|
||||
#[must_use]
|
||||
pub fn deny_key_subject() -> String {
|
||||
format!("$KV.{BUCKET}.{DENY_KEY}")
|
||||
}
|
||||
|
||||
/// The request subject that reads the current deny list.
|
||||
#[must_use]
|
||||
pub fn deny_direct_get_subject() -> String {
|
||||
format!("$JS.API.DIRECT.GET.{STREAM}.{}", deny_key_subject())
|
||||
}
|
||||
|
||||
/// The most entries a list holds, which keeps the value far below the queue's
|
||||
/// message size and the rendered squid configuration small.
|
||||
pub const MAX_ENTRIES: usize = 1000;
|
||||
|
|
@ -89,6 +110,29 @@ impl Entry {
|
|||
}
|
||||
}
|
||||
|
||||
/// One refused destination and who it is refused for.
|
||||
///
|
||||
/// A distinct type from [`Entry`]: a deny is never derived from an agent's
|
||||
/// environment or a linked account, so it carries neither field and nothing
|
||||
/// but an operator's action can put one in the list.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, serde::Serialize, serde::Deserialize)]
|
||||
pub struct DenyEntry {
|
||||
/// A name as [`normalize_domain`] returns it.
|
||||
pub domain: String,
|
||||
/// The agent this entry is for; absent means every agent in the swarm. An
|
||||
/// agent's entry adds to the swarm-wide ones and never lifts one.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub agent: Option<String>,
|
||||
}
|
||||
|
||||
/// The value under [`BUCKET`]/[`DENY_KEY`]. No `deny_unknown_fields`, as for
|
||||
/// [`AllowList`].
|
||||
#[derive(Debug, Clone, Default, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
|
||||
pub struct DenyList {
|
||||
#[serde(default)]
|
||||
pub entries: Vec<DenyEntry>,
|
||||
}
|
||||
|
||||
/// The value under [`BUCKET`]/[`KEY`]. No `deny_unknown_fields`: a field a
|
||||
/// newer controller adds must not cost an older hive the entries it reads.
|
||||
#[derive(Debug, Clone, Default, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
|
||||
|
|
@ -180,6 +224,39 @@ mod tests {
|
|||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_deny_key_is_its_own_subject_of_the_same_bucket() {
|
||||
assert_eq!(deny_key_subject(), "$KV.egress-allow.deny");
|
||||
assert_eq!(
|
||||
deny_direct_get_subject(),
|
||||
"$JS.API.DIRECT.GET.KV_egress-allow.$KV.egress-allow.deny"
|
||||
);
|
||||
assert_ne!(deny_key_subject(), key_subject());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_deny_entry_round_trips_and_tolerates_fields_it_does_not_know() {
|
||||
let list = DenyList {
|
||||
entries: vec![
|
||||
DenyEntry {
|
||||
domain: ".evil.example".to_owned(),
|
||||
agent: None,
|
||||
},
|
||||
DenyEntry {
|
||||
domain: "bad.io".to_owned(),
|
||||
agent: Some("atlas".to_owned()),
|
||||
},
|
||||
],
|
||||
};
|
||||
let wire =
|
||||
r#"{"entries":[{"domain":".evil.example"},{"domain":"bad.io","agent":"atlas"}]}"#;
|
||||
assert_eq!(serde_json::to_string(&list).unwrap(), wire);
|
||||
assert_eq!(serde_json::from_str::<DenyList>(wire).unwrap(), list);
|
||||
let later: DenyList =
|
||||
serde_json::from_str(r#"{"entries":[{"domain":"a.io","later":1}],"x":1}"#).unwrap();
|
||||
assert_eq!(later.entries.len(), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_dns_name_is_kept_lower_cased_with_one_optional_leading_dot() {
|
||||
for (raw, stored) in [
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue