diff --git a/docs/networking/network.md b/docs/networking/network.md index 19cf03f4..bd7d7bb3 100644 --- a/docs/networking/network.md +++ b/docs/networking/network.md @@ -309,6 +309,60 @@ host. It never decrypts TLS, so it holds no signing key. It caches nothing. time or a count. The self-check's users aren't agents, so `deny-canary.invalid` stays a 403. +- **Modes.** An agent is in one of three modes, and the deny list (next bullet) + applies in all of them: + + | mode | the deny list | a name not on the allow list | + | -------------- | ------------- | ---------------------------- | + | `restricted` | blocked | blocked | + | `log` | blocked | allowed and logged | + | `unrestricted` | blocked | allowed | + +- **Deny list.** The swarm's deny list names destinations the proxy refuses + whatever else allows them. The swarm controller keeps it as its own value + (the `deny` key of the `egress-allow` bucket), each entry for every agent or + for one, and serves it at `GET`/`POST`/`DELETE /api/egress/denied-domains` + with the same body as the allow-list (`{"domain": ".example.com", "agent": +"atlas"}`). Write a name as in the allow-list: at least two labels, one + leading dot for the name and its subdomains. The controller refuses IP + addresses, other wildcards, ports and paths. Unlike the allow-list it doesn't + refuse the base list's names: denying one is the point. Every 30 seconds + `hive-c0re` reads the list and renders `services.hyperhive.deploy.hive-controller.squid.denyFile` + (`/var/lib/hive-squid/deny.conf`, `hive-core:squid`, `0640`) for the agents it + runs, and `squid-unrestricted-reload.path` reloads squid when it changes. + - **Precedence.** The configuration includes the file after the destination + denies and before `deny !authed` and every allow, so a denied name is + refused even when the base list, the hive's `allowedDomains`, the swarm's + allow-list, the unrestricted flag or `log` mode would allow it. A swarm-wide + entry names no user, so every client gets a 403 for it, one without a + credential too. An agent's entry also names the agent's squid user, so it + binds that agent: a client that sent no credential gets the usual 407 for + it. An agent's entry only adds to the swarm-wide ones; no entry of one scope + lifts another. + - **Log line.** A request a deny rule refused gets a second log line, + `egress_denied user= method= host= port=

result=`, + in every mode. It differs from the `would_deny` line of `log` mode and + from the `TCP_DENIED/403` line every refusal has, so you can tell a + deny-list hit from a name that isn't on the allow-list. + - **Who writes it.** Only an operator's request to the controller writes the + list. It has no automatic entries: nothing derives a deny from an agent's + API-key environment or a linked account, and the stored type has no such + field. An agent has no queue grant that reaches the key, and the controller's + API sits behind the operator's SSO login: nginx checks it before it hands + a request to the socket, which only nginx and root can open. + - **Empty list.** With no entry `deny.conf` is empty and squid configures as + it did without the list: the configuration gains one `include` line and the + `egress_denied` log lines, and no rule. A hive on a `hive-c0re` that + predates the list never reads the key and keeps its old configuration, so + nothing on its wire changes. + - **Applied.** Each hive reports the revision of the list it last rendered as + `egress_deny_applied` in its status snapshot (see [the hive + snapshot](../swarm/README.md)). `GET /api/egress/denied-domains` answers the + list's `revision` and `behind`, the hives that haven't reported it: an older + `hive-c0re` never does, so it shows there rather than silently allowing the + names. A hive that reads the list reports within about a minute. Reporting + means the file holds the rules; squid reloads moments later. + - **Base list.** `services.hyperhive.deploy.hive-controller.squid.baseAllowedDomains` is open to every authenticated user, whatever its flag: what the agent harness needs off the swarm. The default is the hosts Claude Code requires @@ -419,7 +473,7 @@ host. It never decrypts TLS, so it holds no signing key. It caches nothing. `squid.service`, and `services.hyperhive.otel.journalUnits` forwards that unit and the self-check's to the swarm's log store. A refusal shows as `TCP_DENIED/403` with the client's source address. A request that a `log`-mode agent's rule let - through also logs a `would_deny` line (see _Log mode_). + through also logs a `would_deny` line (see _Log mode_). A request a deny-list rule refused logs an `egress_denied` line (see _Deny list_). - **Self-check.** `squid-selfcheck.timer` runs `squid-selfcheck.service` every minute from the bridge IP, logging in as `selfcheck.squid`: `CONNECT` to `services.hyperhive.deploy.hive-controller.squid.selfcheck.canary` (default: `cache.nixos.org`, diff --git a/docs/swarm/README.md b/docs/swarm/README.md index 71f8bb2b..4b0995ea 100644 --- a/docs/swarm/README.md +++ b/docs/swarm/README.md @@ -99,6 +99,11 @@ of names for what the hive's release can do. Two exist: `tolerant-permissions`, [turning an archetype agent into a custom one](archetypes.md#turn-into-custom) needs. A hive on an older release lists none. +The snapshot also carries `egress_deny_applied`, the revision of the egress deny +list the hive last rendered into its proxy (`0` when the swarm has none). A hive +on an older release, or one that hasn't read the list yet, leaves it out; see +[the egress proxy](../networking/network.md#egress-proxy-squid). + Every row also carries `last_seen_unix` and `age_seconds` if you want to apply your own threshold. The timestamp is the one the queue recorded on arrival, not one the hive put in its own payload. A roster hive's row diff --git a/hive-c0re/src/swarm_status.rs b/hive-c0re/src/swarm_status.rs index 2d9afa41..6f281291 100644 --- a/hive-c0re/src/swarm_status.rs +++ b/hive-c0re/src/swarm_status.rs @@ -122,6 +122,12 @@ pub fn spawn( shutdown.clone(), )); + // The swarm's egress deny list, on the same connection. + tokio::spawn(crate::workers::squid_deny::follow( + client.clone(), + shutdown.clone(), + )); + // The permissions the swarm grants, and the import of the meta repo's. tokio::spawn(crate::workers::swarm_permissions::follow( client.clone(), @@ -375,15 +381,24 @@ struct Snapshot { #[serde(flatten)] readiness: crate::warnings::Readiness, capabilities: &'static [&'static str], + /// The deny-list revision this hive rendered into squid's include + /// (`swarm_wire::hive_status::EGRESS_DENY_APPLIED`); absent until a pass has. + #[serde(skip_serializing_if = "Option::is_none")] + egress_deny_applied: Option, } fn snapshot() -> Snapshot { + snapshot_with(crate::workers::squid_deny::applied()) +} + +fn snapshot_with(egress_deny_applied: Option) -> Snapshot { Snapshot { readiness: crate::warnings::readiness(), capabilities: &[ swarm_wire::hive_status::TOLERANT_PERMISSIONS, swarm_wire::hive_status::NO_CONFIG_CHECKOUT, ], + egress_deny_applied, } } @@ -404,7 +419,7 @@ async fn publish(client: &async_nats::Client, hive: &str) -> Result<()> { #[cfg(test)] mod snapshot_tests { - use super::snapshot; + use super::{snapshot, snapshot_with}; /// The controller reads the capability out of the published bytes with /// the wire type, and the readiness keeps its own top-level fields. @@ -421,4 +436,22 @@ mod snapshot_tests { ); assert!(value.get("status").is_some() && value.get("warnings").is_some()); } + + /// Before a pass has rendered the list the field is absent, as on a hive that + /// predates it; after, the controller reads the revision with the wire type. + #[test] + fn the_snapshot_carries_the_applied_deny_revision_once_there_is_one() { + let fresh = serde_json::to_value(snapshot_with(None)).unwrap(); + assert!( + fresh + .get(swarm_wire::hive_status::EGRESS_DENY_APPLIED) + .is_none() + ); + assert_eq!(swarm_wire::hive_status::egress_deny_applied(&fresh), None); + let rendered = serde_json::to_value(snapshot_with(Some(4))).unwrap(); + assert_eq!( + swarm_wire::hive_status::egress_deny_applied(&rendered), + Some(4) + ); + } } diff --git a/hive-c0re/src/workers/mod.rs b/hive-c0re/src/workers/mod.rs index 4915fdb2..cf839e7f 100644 --- a/hive-c0re/src/workers/mod.rs +++ b/hive-c0re/src/workers/mod.rs @@ -12,6 +12,7 @@ pub mod knowledge; pub mod mcp_sockets; pub mod permissions_import; pub mod squid_allow; +pub mod squid_deny; pub mod squid_users; pub mod swarm_limits; pub mod swarm_permissions; diff --git a/hive-c0re/src/workers/squid_allow.rs b/hive-c0re/src/workers/squid_allow.rs index 889eafbf..0b12a0ff 100644 --- a/hive-c0re/src/workers/squid_allow.rs +++ b/hive-c0re/src/workers/squid_allow.rs @@ -294,7 +294,7 @@ fn seeded(seed: Option<&Path>, logging: &[String]) -> Result { } /// `names` sorted, with each one that another covers dropped. -fn reduced(mut names: Vec) -> Vec { +pub(super) fn reduced(mut names: Vec) -> Vec { names.sort(); names.dedup(); let all = names.clone(); diff --git a/hive-c0re/src/workers/squid_deny.rs b/hive-c0re/src/workers/squid_deny.rs new file mode 100644 index 00000000..a13e026d --- /dev/null +++ b/hive-c0re/src/workers/squid_deny.rs @@ -0,0 +1,452 @@ +//! Squid's deny-list file: the destination names the swarm refuses this hive's +//! agents, whatever allows them. +//! +//! swarm-controller keeps the deny list beside the allow-list +//! ([`swarm_wire::egress_allow::DenyList`], its own key of the same bucket); each +//! entry is for every agent or for one. Every [`INTERVAL`] this reads it and +//! renders [`ENV_DENY_FILE`], squid configuration that +//! `nix/host-modules/hive-squid.nix` `include`s after the destination denies +//! and before `deny !authed` and every allow, so a denied name is refused for +//! every user in every mode. A swarm-wide entry is one `dstdomain` ACL and a +//! `deny` that names no user, so even a client without a credential gets a 403; +//! an agent's entry also names that agent's squid user. Each rule ends in the +//! [`MARK`] annotation, which the proxy logs as its own `egress_denied` line. +//! +//! No entry renders an empty file, which squid reads as no rules, so a hive +//! whose swarm has no deny list configures squid as it did before this existed. +//! +//! The revision of the list last rendered is what [`applied`] answers and the +//! hive's status snapshot carries, so the swarm can see a hive that is behind. +//! A hive that cannot read the list keeps the file it has, and reports nothing +//! it did not render. +//! +//! Every entry is checked again with +//! [`swarm_wire::egress_allow::normalize_domain`], since the names end up in +//! squid's configuration, and skipped when it fails. An entry another entry of +//! its list covers is dropped, because squid ignores the narrower of two +//! overlapping names in one ACL. + +use std::fmt::Write as _; +use std::path::{Path, PathBuf}; +use std::sync::atomic::{AtomicU64, Ordering}; +use std::time::Duration; + +use anyhow::{Context, Result}; +use swarm_queue_client::egress_allow::{self, DenyRead}; +use swarm_wire::egress_allow::{DenyList, covered_by, normalize_domain}; + +use super::squid_allow::reduced; +use super::squid_users::{read_or_empty, write_if_changed}; +use crate::stats::sweep_health::{self, SweepHealth}; + +/// How often the list is read. +pub const INTERVAL: Duration = Duration::from_secs(30); + +/// At [`INTERVAL`] this is ~90 seconds of failing reads. +const FAILURES_BEFORE_WARNING: u32 = 3; + +/// The squid configuration file this renders. Set by `nix/host-modules/hive-c0re` +/// from `services.hyperhive.deploy.hive-controller.squid.denyFile`. +pub const ENV_DENY_FILE: &str = "HIVE_SQUID_DENY_FILE"; + +/// The annotation a deny rule puts on a request, as `key=value`. The `note` ACL +/// in `nix/host-modules/hive-squid.nix` reads the key. +const MARK: &str = "egress_deny=1"; + +/// The ACL that applies [`MARK`]. +const MARK_ACL: &str = "deny_hit_mark"; + +/// The revision last rendered plus one; `0` before the first pass has rendered +/// or confirmed anything. +static APPLIED: AtomicU64 = AtomicU64::new(0); + +/// The revision of the swarm's deny list this hive's file holds, `0` when the +/// swarm has none, or `None` until a pass has said so. +#[must_use] +pub fn applied() -> Option { + APPLIED.load(Ordering::Relaxed).checked_sub(1) +} + +fn set_applied(revision: u64) { + APPLIED.store(revision.saturating_add(1), Ordering::Relaxed); +} + +/// Read the list now and every [`INTERVAL`] until shutdown, when this host has +/// squid's deny file. +pub async fn follow(client: async_nats::Client, mut shutdown: tokio::sync::watch::Receiver) { + let Some(file) = std::env::var_os(ENV_DENY_FILE) + .filter(|v| !v.is_empty()) + .map(PathBuf::from) + else { + return; + }; + let mut health = SweepHealth::new("squid_deny_render", "warn", FAILURES_BEFORE_WARNING); + loop { + match pass(&client, &file).await { + Ok(()) => health.record_ok(), + Err(e) => { + tracing::warn!(error = ?e, "squid deny list: the pass failed"); + let err = format!("{e:#}"); + health.record_err(|ctx| { + let age = ctx.since_last_ok.map_or_else( + || "no success this session".to_owned(), + |d| format!("last ok {} ago", sweep_health::fmt_age(d)), + ); + format!( + "reading the swarm's egress deny list is failing ({} consecutive, \ + {age}) — squid keeps the denies last written: {err}", + ctx.consecutive + ) + }); + } + } + tokio::select! { + () = tokio::time::sleep(INTERVAL) => {} + _ = shutdown.changed() => return, + } + } +} + +async fn pass(client: &async_nats::Client, file: &Path) -> Result<()> { + // An unconnected client does not fail a JetStream request, it hangs on it. + swarm_queue_client::ensure_connected(client)?; + let agents: Vec = crate::lifecycle::agents_for_meta_listing() + .await + .context("enumerating this hive's agents")? + .into_iter() + .map(|spec| spec.name) + .collect(); + let read = egress_allow::read_deny(client) + .await + .map_err(|e| anyhow::anyhow!(e)) + .context("reading the swarm's egress deny list")?; + if let Some(revision) = sync_with(read, &agents, file)? { + set_applied(revision); + } + Ok(()) +} + +/// [`pass`] with the read and the agents given. The file is only written when +/// its rendering differs from what it holds. Answers the revision the file now +/// holds, or `None` when it cannot say. +fn sync_with(read: DenyRead, agents: &[String], file: &Path) -> Result> { + match read { + DenyRead::Current { list, revision } => { + let rendered = render(&list, agents); + if write_if_changed(file, &rendered.text)? { + tracing::info!( + file = %file.display(), revision, + "squid deny list: {} swarm, {} agent entries", rendered.swarm, rendered.agent + ); + } + if !rendered.skipped.is_empty() { + tracing::warn!( + skipped = ?rendered.skipped, + "squid deny list: entries the swarm holds were skipped as invalid" + ); + } + Ok(Some(revision)) + } + // Nothing to read says nothing about what the swarm wants, so a file + // that holds denies stays. An empty file is the true "no list". + DenyRead::NotSet => Ok(read_or_empty(file)?.is_empty().then_some(0)), + } +} + +struct Rendered { + text: String, + /// Swarm-wide names written. + swarm: usize, + /// Names written across this hive's agents. + agent: usize, + /// Entries left out because they are not valid names or agents. + skipped: Vec, +} + +/// The squid configuration for `list`, for the agents this hive runs. Empty for +/// no entry that applies. +/// +/// The destination ACL comes first in each rule: squid checks ACLs left to +/// right, so a swarm-wide rule never reaches a `proxy_auth` ACL, which would +/// answer an anonymous client 407 instead of the 403. +fn render(list: &DenyList, agents: &[String]) -> Rendered { + let mut skipped = Vec::new(); + let mut swarm_wide = Vec::new(); + let mut per_agent: std::collections::BTreeMap<&str, Vec> = + std::collections::BTreeMap::new(); + for entry in &list.entries { + let Ok(domain) = normalize_domain(&entry.domain) else { + skipped.push(entry.domain.clone()); + continue; + }; + match entry.agent.as_deref() { + None => swarm_wide.push(domain), + Some(agent) => { + if hive_types::Ident::parse(agent).is_err() { + skipped.push(format!("{} for {agent}", entry.domain)); + } else if let Some(mine) = agents.iter().find(|a| *a == agent) { + per_agent.entry(mine.as_str()).or_default().push(domain); + } + } + } + } + let swarm_wide = reduced(swarm_wide); + + let mut rules = String::new(); + if !swarm_wide.is_empty() { + let _ = write!( + rules, + "acl deny_swarm_dst dstdomain {}\n\ + http_access deny deny_swarm_dst {MARK_ACL}\n", + swarm_wide.join(" ") + ); + } + let mut agent_count = 0; + for (agent, names) in per_agent { + let names = reduced( + names + .into_iter() + .filter(|d| !covered_by(d, &swarm_wide)) + .collect(), + ); + if names.is_empty() { + continue; + } + agent_count += names.len(); + let _ = write!( + rules, + "acl deny_agent_{agent}_dst dstdomain {}\n\ + acl deny_agent_{agent} proxy_auth {agent}\n\ + http_access deny deny_agent_{agent}_dst deny_agent_{agent} {MARK_ACL}\n", + names.join(" ") + ); + } + let text = if rules.is_empty() { + String::new() + } else { + format!("acl {MARK_ACL} annotate_transaction {MARK}\n{rules}") + }; + Rendered { + text, + swarm: swarm_wide.len(), + agent: agent_count, + skipped, + } +} + +#[cfg(test)] +mod tests { + use super::*; + use swarm_wire::egress_allow::DenyEntry; + + fn list(entries: &[(&str, Option<&str>)]) -> DenyList { + DenyList { + entries: entries + .iter() + .map(|(d, a)| DenyEntry { + domain: (*d).to_owned(), + agent: a.map(str::to_owned), + }) + .collect(), + } + } + + fn agents(names: &[&str]) -> Vec { + names.iter().map(|n| (*n).to_owned()).collect() + } + + const MARK_LINE: &str = "acl deny_hit_mark annotate_transaction egress_deny=1\n"; + + /// The one place a rule's shape is pinned: the destination ACL before any + /// user ACL, and the annotation last. + #[test] + fn a_swarm_wide_name_is_one_destination_rule_that_names_no_user() { + let r = render( + &list(&[("evil.com", None), (".bad.io", None), ("EVIL.com", None)]), + &agents(&["atlas"]), + ); + assert_eq!( + r.text, + format!( + "{MARK_LINE}acl deny_swarm_dst dstdomain .bad.io evil.com\n\ + http_access deny deny_swarm_dst deny_hit_mark\n" + ) + ); + assert!(!r.text.contains("proxy_auth"), "no user, so no 407"); + assert_eq!((r.swarm, r.agent), (2, 0)); + } + + #[test] + fn an_agents_name_is_a_destination_then_user_rule_and_other_hives_agents_are_left_out() { + let r = render( + &list(&[ + ("pypi.org", Some("atlas")), + ("crates.io", Some("atlas")), + ("github.com", Some("iris")), + ("github.com", Some("../argus")), + ]), + &agents(&["atlas", "argus"]), + ); + assert_eq!( + r.text, + format!( + "{MARK_LINE}acl deny_agent_atlas_dst dstdomain crates.io pypi.org\n\ + acl deny_agent_atlas proxy_auth atlas\n\ + http_access deny deny_agent_atlas_dst deny_agent_atlas deny_hit_mark\n" + ) + ); + assert_eq!((r.swarm, r.agent), (0, 2)); + assert_eq!(r.skipped, ["github.com for ../argus"]); + assert!(!r.text.contains("iris"), "another hive's agent"); + assert!(!r.text.contains("argus"), "an invalid name"); + } + + /// An agent's entry adds to the swarm-wide rule and never replaces it: the + /// swarm-wide rule renders whole whatever the agent lists, and a name it + /// already covers is not repeated for the agent. + #[test] + fn an_agents_entry_adds_to_the_swarm_wide_rule_and_never_narrows_it() { + let r = render( + &list(&[ + (".evil.com", None), + ("api.evil.com", Some("atlas")), + ("bad.io", Some("atlas")), + ]), + &agents(&["atlas"]), + ); + assert!( + r.text.contains( + "acl deny_swarm_dst dstdomain .evil.com\nhttp_access deny deny_swarm_dst" + ) + ); + assert!(r.text.contains("deny_agent_atlas_dst dstdomain bad.io\n")); + assert!(!r.text.contains("api.evil.com"), "{}", r.text); + assert_eq!((r.swarm, r.agent), (1, 1)); + } + + /// No entry renders no byte, which is what keeps a hive with no deny list + /// configured as it was. + #[test] + fn no_entry_that_applies_renders_an_empty_file() { + assert_eq!(render(&DenyList::default(), &agents(&["atlas"])).text, ""); + // An agent this hive does not run, and an invalid name, render nothing. + let r = render( + &list(&[("evil.com", Some("iris")), ("1.2.3.4", None)]), + &agents(&["atlas"]), + ); + assert_eq!(r.text, ""); + assert_eq!(r.skipped, ["1.2.3.4"]); + // Control: the same list with a name that applies is not empty. + assert!( + !render(&list(&[("evil.com", None)]), &agents(&[])) + .text + .is_empty() + ); + } + + #[test] + fn an_entry_that_is_not_a_name_never_reaches_squid_configuration() { + let r = render( + &list(&[ + ("evil.com", None), + ("*.example.com", Some("atlas")), + ("example.com\nhttp_access allow all", Some("atlas")), + ("example.com\nhttp_access allow all", None), + ]), + &agents(&["atlas"]), + ); + assert!(!r.text.contains("allow"), "{}", r.text); + assert!(r.text.contains("dstdomain evil.com\n")); + assert_eq!(r.skipped.len(), 3); + } + + /// The value swarm-controller's `egress_deny` route publishes, byte for byte + /// as its own test pins it, renders the rules for the agents named in it. + #[test] + fn the_controllers_published_value_renders_the_rules_it_names() { + let wire = r#"{"entries":[{"domain":"bad.io","agent":"atlas"},{"domain":"evil.com"},{"domain":"evil.com","agent":"atlas"}]}"#; + let decoded: DenyList = serde_json::from_str(wire).expect("decodes"); + let r = render(&decoded, &agents(&["atlas"])); + assert!(r.text.contains("acl deny_swarm_dst dstdomain evil.com\n")); + assert!( + r.text + .contains("acl deny_agent_atlas_dst dstdomain bad.io\n") + ); + assert_eq!((r.swarm, r.agent), (1, 1)); + } + + struct Dir(tempfile::TempDir); + + impl Dir { + fn file(&self) -> PathBuf { + self.0.path().join("deny.conf") + } + } + + fn dir() -> Dir { + Dir(tempfile::tempdir().expect("tempdir")) + } + + fn text(d: &Dir) -> String { + std::fs::read_to_string(d.file()).expect("the file is there") + } + + fn current(entries: &[(&str, Option<&str>)], revision: u64) -> DenyRead { + DenyRead::Current { + list: list(entries), + revision, + } + } + + #[test] + fn the_file_follows_the_list_and_the_claimed_revision_follows_the_file() { + let d = dir(); + let mine = agents(&["atlas"]); + let sync = |read| sync_with(read, &mine, &d.file()).expect("renders"); + + assert_eq!(sync(current(&[("evil.com", None)], 7)), Some(7)); + assert!(text(&d).contains("dstdomain evil.com")); + + // A second pass of the same list writes nothing. + let ino = std::os::unix::fs::MetadataExt::ino(&std::fs::metadata(d.file()).unwrap()); + assert_eq!(sync(current(&[("evil.com", None)], 7)), Some(7)); + assert_eq!( + std::os::unix::fs::MetadataExt::ino(&std::fs::metadata(d.file()).unwrap()), + ino + ); + + // No answer keeps the denies in force and claims no revision. + assert_eq!(sync(DenyRead::NotSet), None); + assert!(text(&d).contains("dstdomain evil.com"), "denies are kept"); + + // The list emptied: the file is empty and the revision moves. + assert_eq!(sync(current(&[], 9)), Some(9)); + assert_eq!(text(&d), ""); + + // An empty file with nothing to read is the true "no list": revision 0. + assert_eq!(sync(DenyRead::NotSet), Some(0)); + } + + /// The only test that touches the process-wide value. + #[test] + fn nothing_is_applied_until_a_pass_says_so() { + assert_eq!(applied(), None); + set_applied(0); + assert_eq!(applied(), Some(0)); + set_applied(12); + assert_eq!(applied(), Some(12)); + } + + #[test] + fn the_file_is_group_readable() { + use std::os::unix::fs::PermissionsExt as _; + let d = dir(); + sync_with(current(&[("evil.com", None)], 1), &agents(&[]), &d.file()).expect("renders"); + let mode = std::fs::metadata(d.file()) + .expect("stat") + .permissions() + .mode() + & 0o777; + assert_eq!(mode, 0o640); + } +} diff --git a/nix/host-modules/hive-c0re/environment.nix b/nix/host-modules/hive-c0re/environment.nix index 1d4976b3..6fd4e965 100644 --- a/nix/host-modules/hive-c0re/environment.nix +++ b/nix/host-modules/hive-c0re/environment.nix @@ -299,6 +299,9 @@ in HIVE_SQUID_ALLOWED_SEED_FILE = config.services.hyperhive.deploy.hive-controller.squid.allowedSeedFile; HIVE_SQUID_ALWAYS_ALLOWED = lib.concatStringsSep " " config.services.hyperhive.deploy.hive-controller.squid.alwaysAllowed; + # The swarm's egress deny list, which `workers::squid_deny` renders into the + # include squid reads before every allow. + HIVE_SQUID_DENY_FILE = config.services.hyperhive.deploy.hive-controller.squid.denyFile; # The egress proxy facts every container is handed (`HiveFacts` in # `hive-priv-sock`). They sit with the users file above because that is what # makes squid's credential check and per-agent policy real on this host: an diff --git a/nix/host-modules/hive-squid.nix b/nix/host-modules/hive-squid.nix index db122586..6120b8bb 100644 --- a/nix/host-modules/hive-squid.nix +++ b/nix/host-modules/hive-squid.nix @@ -10,6 +10,8 @@ # the swarm lists for that agent alone. A third kind, set per agent # (`internet_log`), reaches the same names but is also let through to any other # destination the deny rules leave; the request is logged as `would_deny`. +# The swarm's deny list (rendered by hive-c0re into `denyFile`) is refused for +# all three before any allow, and logged as `egress_denied`. # An agent sends its credential # once it has fetched it (../agent-modules/egress-proxy.nix); the exporter, # journal forwarding and the self-check below show the proxy works. @@ -89,6 +91,10 @@ let # `deny !allowed_dst`, so it sees the same denies and only turns that last # refusal into an allow. The `dst` denies cost an unlisted name a lookup on # the host's resolver before it is refused. + # The swarm's deny list (`denyFile`) sits right after those destination + # denies and before `deny !authed`, so a denied name is refused before any + # credential check and before every allow, in every mode: its rules name the + # destination first, and a swarm-wide one names no user at all. squidConf = '' http_port ${listen} pid_filename /run/squid/squid.pid @@ -110,6 +116,14 @@ let acl would_denied note would_deny 1 access_log syslog:daemon.info logformat=would_deny would_denied + # A request one of the deny list's rules (hive-c0re's `denyFile`, below) + # refused carries the annotation `egress_deny=1` and gets a line of its own, + # distinct from `would_deny` and from the 403 line every refusal has, in + # every mode. + logformat egress_denied egress_denied user=%[un method=%rm host=%>rd port=%>rP result=%Ss/%03>Hs + acl egress_denied_hit note egress_deny 1 + access_log syslog:daemon.info logformat=egress_denied egress_denied_hit + # The helper reads the users file on every request and reloads it when its # mtime moves. It exits when the file is missing, which is why the file is # created before squid starts. A positive answer is cached for a minute, so @@ -153,6 +167,7 @@ let http_access deny to_linklocal http_access deny to_bridge http_access deny to_private + include ${cfg.denyFile} http_access deny !authed ${lib.optionalString (base != [ ]) "http_access allow base_dst"} include ${cfg.agentAllowFile} @@ -162,9 +177,9 @@ let http_access deny all ''; - # `squid -k parse` on the configuration with the include pointing at an empty + # `squid -k parse` on the configuration with the includes pointing at an empty # file. The nixpkgs module runs the same parse on the real text, which - # cannot open the include in the build sandbox. + # cannot open the includes in the build sandbox. squidConfCheck = pkgs.runCommand "squid-conf-check" { @@ -173,7 +188,7 @@ let '' squid -k parse -f ${ pkgs.writeText "squid.conf" ( - builtins.replaceStrings [ cfg.agentAllowFile ] [ "/dev/null" ] squidConf + builtins.replaceStrings [ cfg.agentAllowFile cfg.denyFile ] [ "/dev/null" "/dev/null" ] squidConf ) } touch $out @@ -381,6 +396,24 @@ in ''; }; + denyFile = lib.mkOption { + type = lib.types.path; + default = "${builtins.dirOf cfg.usersFile}/deny.conf"; + readOnly = true; + defaultText = lib.literalMD "`deny.conf`, in the directory of `usersFile`"; + description = '' + Squid configuration that hive-c0re generates from the swarm's deny list + (`/api/egress/denied-domains`) and the main configuration includes after + the destination denies and before `deny !authed` and every allow: a + `http_access deny` for the swarm-wide names, and one per agent of this + hive the swarm lists names for. A denied name is refused for every user + in every egress mode, whether or not an allow rule covers it. The file + is empty, and squid configured as without it, until the swarm lists a + name. Squid is reloaded when it changes. Owned by `hive-core` and readable + by the `squid` group. + ''; + }; + allowedSeedFile = lib.mkOption { type = lib.types.path; default = seedFile; @@ -481,6 +514,7 @@ in "f ${cfg.unrestrictedFile} 0640 hive-core squid - -" "C ${cfg.allowedFile} 0640 hive-core squid - ${seedFile}" "f ${cfg.agentAllowFile} 0640 hive-core squid - -" + "f ${cfg.denyFile} 0640 hive-core squid - -" ]; # Squid reads the unrestricted list, the allow-list and the per-agent include @@ -494,6 +528,7 @@ in cfg.unrestrictedFile cfg.allowedFile cfg.agentAllowFile + cfg.denyFile ]; }; diff --git a/nix/module-eval/hive-otel.nix b/nix/module-eval/hive-otel.nix index 1205bbeb..d01be083 100644 --- a/nix/module-eval/hive-otel.nix +++ b/nix/module-eval/hive-otel.nix @@ -636,6 +636,58 @@ let && builtins.elem ''acl unrestricted proxy_auth "${list}"'' lines && list == "/var/lib/hive-squid/unrestricted"; } + { + # The swarm's deny list is an include that precedes every allow and the + # credential check, so a denied name is refused in every mode and for a + # client that sent no credential. Control: the include and the allows are + # found, so a position compared below is not a lookup of nothing. + name = "squid includes the swarm's deny list after the destination denies and before the credential check and every allow"; + ok = + let + sq = hiveOtel.services.hyperhive.deploy.hive-controller.squid; + lines = lib.splitString "\n" hiveOtel.services.squid.configText; + at = l: lib.lists.findFirstIndex (x: x == l) null lines; + deny = at "include ${sq.denyFile}"; + allows = map at [ + "http_access allow base_dst" + "include ${sq.agentAllowFile}" + "http_access allow unrestricted" + "http_access allow allowed_dst" + ]; + in + deny != null + && sq.denyFile == "/var/lib/hive-squid/deny.conf" + && !(lib.elem null allows) + && lib.all (a: deny < a) allows + && at "http_access deny to_private" < deny + && at "http_access deny ip_v4_host" < deny + && deny < at "http_access deny !authed" + # Two includes: this one and the per-agent allow rules. + && lib.length (lib.filter (l: lib.hasPrefix "include " l) lines) == 2; + } + { + # What the deny list adds to the configuration is the include and the + # lines that log a hit; with the file empty, which tmpfiles creates, squid + # has no rule it did not have. Control: the lines are found, so the + # equality below is not against an empty filter. + name = "squid's deny list adds one include and the log lines, and its file exists empty before squid starts"; + ok = + let + sq = hiveOtel.services.hyperhive.deploy.hive-controller.squid; + conf = hiveOtel.services.squid.configText; + mentions = lib.filter ( + l: !(lib.hasPrefix "#" l) && (lib.hasInfix "egress_den" l || lib.hasInfix sq.denyFile l) + ) (lib.splitString "\n" conf); + in + mentions == [ + "logformat egress_denied egress_denied user=%[un method=%rm host=%>rd port=%>rP result=%Ss/%03>Hs" + "acl egress_denied_hit note egress_deny 1" + "access_log syslog:daemon.info logformat=egress_denied egress_denied_hit" + "include ${sq.denyFile}" + ] + && builtins.elem "f ${sq.denyFile} 0640 hive-core squid - -" hiveOtel.systemd.tmpfiles.rules + && hiveOtelAllLocal.systemd.services.hive-c0re.environment.HIVE_SQUID_DENY_FILE == sq.denyFile; + } { # The list is read when squid parses its configuration, so a change to # the file reloads squid. Control: the watched path is the list's own. @@ -651,6 +703,7 @@ let list hiveOtel.services.hyperhive.deploy.hive-controller.squid.allowedFile hiveOtel.services.hyperhive.deploy.hive-controller.squid.agentAllowFile + hiveOtel.services.hyperhive.deploy.hive-controller.squid.denyFile ] && hiveOtel.systemd.services.squid-unrestricted-reload.startLimitIntervalSec == 0 && builtins.elem "paths.target" path.wantedBy diff --git a/nix/vm-tests/hive-squid.nix b/nix/vm-tests/hive-squid.nix index 4607042c..aa5051df 100644 --- a/nix/vm-tests/hive-squid.nix +++ b/nix/vm-tests/hive-squid.nix @@ -253,6 +253,7 @@ pkgs.testers.runNixOSTest { LIST = "/var/lib/hive-squid/unrestricted" ALLOWED = "/var/lib/hive-squid/allowed" AGENT_ALLOW = "/var/lib/hive-squid/agent-allow.conf" + DENY = "/var/lib/hive-squid/deny.conf" UNRESTRICTED = ["uma", "selfcheck-unrestricted.squid"] start_all() @@ -304,6 +305,14 @@ pkgs.testers.runNixOSTest { f"&& chmod 640 {path}.new && mv {path}.new {path}" ) + def write_deny(body): + # What hive-c0re does for the deny list: written aside and renamed into + # place, in squid's group; the reload is squid-unrestricted-reload's. + proxy.succeed( + f"printf '%s' {shlex.quote(body)} > {DENY}.new && chown hive-core:squid {DENY}.new " + f"&& chmod 640 {DENY}.new && mv {DENY}.new {DENY}" + ) + def wait_status(host, cred, want): # The reload that follows a list write is asynchronous. got = None @@ -328,6 +337,8 @@ pkgs.testers.runNixOSTest { assert proxy.succeed(f"stat -c '%U:%G %a' {ALLOWED}").strip() == "hive-core:squid 640" assert proxy.succeed(f"stat -c '%U:%G %a' {AGENT_ALLOW}").strip() == "hive-core:squid 640" assert proxy.succeed(f"stat -c %s {AGENT_ALLOW}").strip() == "0" + assert proxy.succeed(f"stat -c '%U:%G %a' {DENY}").strip() == "hive-core:squid 640" + assert proxy.succeed(f"stat -c %s {DENY}").strip() == "0", "no deny list, no rule" # The allow-list file starts as the hive's own allowedDomains. proxy.succeed(f"grep -Fx forge.t.local {ALLOWED}") assert connect("forge.t.local", ATLAS) == "407", "no such user" @@ -573,6 +584,53 @@ pkgs.testers.runNixOSTest { write_allow(seeded, "") wait_status("other.t.local", RUTH, "403") + with subtest("the swarm's deny list is refused before any allow, in every mode, with or without a credential, and logged as egress_denied"): + # What hive-c0re renders: `forge.t.local` and `other.t.local` for every + # agent, `api.t.local` (a base-list name) for atlas alone. `ruth` is in + # log mode, so her rule follows. The names are allowed or reachable + # without the list, which the controls establish first. + assert connect("forge.t.local", ATLAS) == "200", "control: allowed by the hive" + assert connect("forge.t.local", UMA) == "200", "control: allowed, unrestricted" + assert connect("other.t.local", UMA) == "200", "control: unrestricted reaches it" + assert connect("api.t.local", ATLAS) == "200", "control: base list" + write_allow( + seeded, + "# source: swarm\n" + "acl log_mark annotate_transaction would_deny=1\n" + "acl agent_log_ruth proxy_auth ruth\n" + "http_access allow agent_log_ruth !allowed_dst log_mark\n", + ) + wait_status("other.t.local", RUTH, "200") + write_deny( + "acl deny_hit_mark annotate_transaction egress_deny=1\n" + "acl deny_swarm_dst dstdomain forge.t.local other.t.local\n" + "http_access deny deny_swarm_dst deny_hit_mark\n" + "acl deny_agent_atlas_dst dstdomain api.t.local\n" + "acl deny_agent_atlas proxy_auth atlas\n" + "http_access deny deny_agent_atlas_dst deny_agent_atlas deny_hit_mark\n" + ) + wait_status("forge.t.local", ATLAS, "403") + assert connect("forge.t.local", RUTH) == "403", "a listed name, log mode" + assert connect("forge.t.local", UMA) == "403", "a listed name, unrestricted" + assert connect("other.t.local", UMA) == "403", "unrestricted" + assert connect("other.t.local", RUTH) == "403", "log mode" + assert connect("forge.t.local") == "403", "no credential: refused, not challenged" + assert connect("api.t.local", ATLAS) == "403", "a base-list name, denied for atlas" + assert connect("api.t.local", RUTH) == "200", "control: another agent keeps the base name" + assert connect("api.t.local") == "407", "control: a per-agent deny does not answer a client it cannot name" + proxy.wait_until_succeeds( + "journalctl -u squid.service -o cat | grep -E " + "'egress_denied user=atlas method=CONNECT host=api\\.t\\.local port=443 result=TCP_DENIED/403'" + ) + assert squid_saw(r"egress_denied user=- method=CONNECT host=forge\.t\.local port=443 "), "the anonymous client's hit" + assert not squid_saw(r"egress_denied .* host=loop\.t\.local"), "another refusal is not a deny-list hit" + assert not squid_saw(r"egress_denied user=ruth .* host=api\.t\.local"), "another agent's name is not a hit" + write_deny("") + write_allow(seeded, "") + wait_status("forge.t.local", ATLAS, "200") + assert connect("api.t.local", ATLAS) == "200", "the list emptied" + assert connect("other.t.local", UMA) == "200", "the list emptied, unrestricted" + with subtest("squid starts on a missing list, treats every user as restricted, and recovers when it appears"): proxy.succeed(f"rm {LIST}") proxy.succeed("systemctl restart squid.service") diff --git a/swarm-controller-api/src/egress.rs b/swarm-controller-api/src/egress.rs index 783b8ffc..acb5179b 100644 --- a/swarm-controller-api/src/egress.rs +++ b/swarm-controller-api/src/egress.rs @@ -1,5 +1,5 @@ -//! `/api/egress/allowed-domains` — the destinations the swarm lets agents -//! reach through their hive's egress proxy. +//! `/api/egress/allowed-domains` and `/api/egress/denied-domains` — the destinations the swarm lets +//! agents reach through their hive's egress proxy, and the ones it refuses. use serde::{Deserialize, Serialize}; @@ -33,6 +33,40 @@ pub struct AllowedDomains { pub entries: Vec, } +/// One refused destination and who it is refused for. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[cfg_attr(feature = "schema", derive(utoipa::ToSchema))] +#[serde(deny_unknown_fields)] +pub struct DeniedDomain { + /// A DNS name written as in [`AllowedDomain::domain`]: at least two + /// labels, lower-cased, one leading dot for the name and its subdomains. + #[cfg_attr(feature = "schema", schema(example = ".example.com"))] + pub domain: String, + /// The agent this entry is for. Absent means every agent in the swarm. An + /// agent's entry adds to the swarm-wide ones and never lifts one. + #[serde(default, skip_serializing_if = "Option::is_none")] + #[cfg_attr(feature = "schema", schema(example = "atlas"))] + pub agent: Option, +} + +/// The swarm's deny list: every entry, swarm-wide and per agent, sorted, and +/// whether every hive has taken it. +#[derive(Debug, Default, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[cfg_attr(feature = "schema", derive(utoipa::ToSchema))] +pub struct DeniedDomains { + pub entries: Vec, + /// The revision of the stored list, `0` when nobody has written one. A + /// hive reports the revision it last rendered into its proxy. + #[serde(default)] + pub revision: u64, + /// The hives whose proxy is not known to refuse these names yet: they have + /// not reported the list's revision (a hive that predates the deny list + /// never does, a new one takes a minute or so), or have not reported at + /// all. Empty when the list is empty and no hive reports otherwise. + #[serde(default)] + pub behind: Vec, +} + #[cfg(test)] mod tests { use super::*; @@ -60,4 +94,23 @@ mod tests { ); assert!(serde_json::from_str::(r#"{"domain":"a.io","hive":"x"}"#).is_err()); } + + #[test] + fn a_denied_entry_has_no_derived_flag_and_refuses_unknown_fields() { + let swarm_wide = DeniedDomain { + domain: ".example.com".to_owned(), + agent: None, + }; + let wire = r#"{"domain":".example.com"}"#; + assert_eq!(serde_json::to_string(&swarm_wide).unwrap(), wire); + assert_eq!( + serde_json::from_str::(wire).unwrap(), + swarm_wide + ); + // A deny is never automatic, so the allow list's `derived` is not a field. + assert!( + serde_json::from_str::(r#"{"domain":"a.io","derived":true}"#).is_err() + ); + assert!(serde_json::from_str::(r#"{"domain":"a.io","hive":"x"}"#).is_err()); + } } diff --git a/swarm-controller/README.md b/swarm-controller/README.md index bc603edf..fb4e59f9 100644 --- a/swarm-controller/README.md +++ b/swarm-controller/README.md @@ -68,7 +68,10 @@ The swarm's control plane. The swarm UI and `swarmctl` are its clients. (`GET`/`POST`/`DELETE /api/egress/allowed-domains`); every hive reads it and renders the proxy's allow-list files, which add to the names the hive's own configuration lists. It also keeps a `derived` entry per agent for the host - of its stored backend environment's `ANTHROPIC_BASE_URL` → + of its stored backend environment's `ANTHROPIC_BASE_URL`. The deny list is a second + value of that bucket (`GET`/`POST`/`DELETE /api/egress/denied-domains`) with no + derived entries; every hive renders it into an include the proxy checks before + any allow, and reports the revision it rendered in its status snapshot → [`docs/networking/network.md`](../docs/networking/network.md#egress-proxy-squid). - **Agent instructions** — each agent's `AGENTS.md` as its harness last published it into the `agents-md` KV bucket diff --git a/swarm-controller/src/egress_deny.rs b/swarm-controller/src/egress_deny.rs new file mode 100644 index 00000000..54947180 --- /dev/null +++ b/swarm-controller/src/egress_deny.rs @@ -0,0 +1,755 @@ +//! `/api/egress/denied-domains` — the destinations the swarm refuses agents +//! through their hive's egress proxy, whatever else allows them. +//! +//! The list is one value under the `deny` key of the `egress-allow` bucket +//! (`swarm_wire::egress_allow`), which this daemon alone writes, beside the +//! allow-list's key. Every hive reads it and renders the entries for the agents +//! it runs: a swarm-wide entry is refused for everyone, an entry with an agent +//! for that agent alone. The proxy checks the list before any allow rule, so an +//! entry beats the base list, the hive's own names, the allow-list, the +//! unrestricted flag and `log` mode alike, and an agent's entry can only add to +//! the swarm-wide ones. +//! +//! Only an operator's request writes an entry. There is no derived entry: the +//! stored type has no `derived` field, and nothing in this daemon reads an +//! agent's environment or a linked account into this list. +//! +//! Each name goes into the proxy's configuration, so it is checked with +//! `swarm_wire::egress_allow::normalize_domain` here and again by the hive. +//! The base list is not consulted: refusing a name the proxy otherwise always +//! allows is the point of the list. +//! +//! A hive reports the revision of the list it rendered in its status snapshot +//! (`swarm_wire::hive_status::EGRESS_DENY_APPLIED`); [`behind`] names the hives +//! whose report does not match, which the reads show as `behind`. + +use axum::Json; +use axum::extract::State; +use axum::http::StatusCode; +use problem_details::ProblemDetails; +use swarm_controller_api::egress::{DeniedDomain, DeniedDomains}; +use swarm_queue_client::egress_allow; +use swarm_wire::egress_allow::{DENY_KEY, DenyEntry, DenyList, MAX_ENTRIES, normalize_domain}; +use swarm_wire::hive_status::egress_deny_applied; + +use crate::routes::{AppState, error_problem, wanted_error_status}; +use crate::schedules::{live, queue, read_or_default, refused, write_revision}; +use crate::status::{Freshness, HiveStatus}; + +/// How many times a losing writer re-reads and re-applies, as in +/// `egress_allow::MAX_ATTEMPTS`. +const MAX_ATTEMPTS: usize = 5; + +/// The entry to store for `req`, or why it is refused. +pub(crate) fn checked(req: &DeniedDomain) -> Result { + let domain = normalize_domain(&req.domain)?; + let agent = req + .agent + .as_deref() + .map(|a| { + hive_types::Ident::parse(a) + .map(hive_types::Ident::into_string) + .map_err(|reason| format!("agent {a:?}: {reason}")) + }) + .transpose()?; + Ok(DenyEntry { domain, agent }) +} + +/// The hives among `rows` whose proxy is not known to refuse the list at +/// `revision` yet. +/// +/// A hive that reports a revision is behind when it differs. A hive that +/// reports none (it predates the field, has not read the list, or has never +/// reported) is behind only while the list holds an entry: it cannot be +/// enforcing a name it was never told, but an empty list asks nothing of it. +/// A hive the roster does not list is left out. +pub(crate) fn behind(rows: &[HiveStatus], revision: u64, list_is_empty: bool) -> Vec { + rows.iter() + .filter(|row| row.freshness != Freshness::Unknown) + .filter( + |row| match row.snapshot.as_ref().and_then(egress_deny_applied) { + Some(applied) => applied != revision, + None => !list_is_empty, + }, + ) + .map(|row| row.name.clone()) + .collect() +} + +/// The list as the API answers it. +fn view(list: DenyList, revision: u64, behind: Vec) -> DeniedDomains { + let mut entries = list.entries; + entries.sort(); + DeniedDomains { + entries: entries + .into_iter() + .map(|e| DeniedDomain { + domain: e.domain, + agent: e.agent, + }) + .collect(), + revision, + behind, + } +} + +/// Which hives are behind `list` at `revision`. A status read that fails names +/// every hive, since nothing then confirms any. +async fn behind_hives(state: &AppState, revision: u64, list: &DenyList) -> Vec { + let Some(reader) = state.status.as_ref() else { + return Vec::new(); + }; + match reader + .view(&state.hives, std::time::SystemTime::now()) + .await + { + Ok(rows) => behind(&rows, revision, list.entries.is_empty()), + Err(e) => { + tracing::warn!(error = %format!("{e:#}"), "egress deny list: reading the hives' status failed"); + state.hives.iter().map(|h| h.name.clone()).collect() + } + } +} + +fn queue_problem(e: &anyhow::Error) -> ProblemDetails { + tracing::warn!(error = %format!("{e:#}"), "egress deny list: queue failure"); + error_problem(wanted_error_status(e), &format!("{e:#}")) +} + +async fn open_or_create( + client: &async_nats::Client, +) -> Result { + egress_allow::open_or_create(client) + .await + .map_err(|e| queue_problem(&anyhow::Error::new(e))) +} + +#[utoipa::path( + get, + path = "/api/egress/denied-domains", + responses( + (status = 200, description = "every entry, swarm-wide and per agent, sorted, the \ + stored revision, and the hives that have not reported rendering it; empty when \ + the swarm has not set a list", + body = DeniedDomains), + (status = 503, description = "no swarm queue is configured on this host, or it is \ + not connected (problem+json)", body = String), + (status = 500, description = "the list could not be read (problem+json)", + body = String), + ), + tag = "egress" +)] +pub(crate) async fn get_denied_domains( + State(state): State, +) -> Result, ProblemDetails> { + let client = queue(&state).map_err(refused)?; + let Some(store) = egress_allow::open(&client) + .await + .map_err(|e| queue_problem(&anyhow::Error::new(e)))? + else { + return Ok(Json(DeniedDomains::default())); + }; + let (list, revision, _) = read_or_default::(&store, DENY_KEY) + .await + .map_err(|e| queue_problem(&e))?; + let revision = revision.unwrap_or(0); + let behind = behind_hives(&state, revision, &list).await; + Ok(Json(view(list, revision, behind))) +} + +/// What changing a list came to. +enum Changed { + /// Nothing to write. + Same(DenyList, u64), + /// The list to write. + New(DenyList), +} + +/// `list` with `entry` added, or why it can't take another entry. +fn added(mut list: DenyList, entry: &DenyEntry, revision: u64) -> Result { + if list.entries.contains(entry) { + return Ok(Changed::Same(list, revision)); + } + if list.entries.len() >= MAX_ENTRIES { + return Err(format!( + "the list holds {MAX_ENTRIES} entries, which is the most it takes" + )); + } + list.entries.push(entry.clone()); + list.entries.sort(); + Ok(Changed::New(list)) +} + +/// `list` without `entry`. +fn removed(mut list: DenyList, entry: &DenyEntry, revision: u64) -> Changed { + if !list.entries.contains(entry) { + return Changed::Same(list, revision); + } + list.entries.retain(|e| e != entry); + Changed::New(list) +} + +/// Re-read the list and write `change` of it, retrying a lost write, and answer +/// the list as it stands after. +async fn apply( + state: &AppState, + client: &async_nats::Client, + what: &str, + change: impl Fn(DenyList, u64) -> Result, +) -> Result, ProblemDetails> { + let store = open_or_create(client).await?; + for _ in 0..MAX_ATTEMPTS { + let (list, revision, _) = read_or_default::(&store, DENY_KEY) + .await + .map_err(|e| queue_problem(&e))?; + let list = match change(list, revision.unwrap_or(0)) { + Ok(Changed::Same(list, revision)) => { + let behind = behind_hives(state, revision, &list).await; + return Ok(Json(view(list, revision, behind))); + } + Ok(Changed::New(list)) => list, + Err(why) => return Err(error_problem(StatusCode::BAD_REQUEST, &why)), + }; + if let Some(written) = write_revision(&store, DENY_KEY, &list, revision) + .await + .map_err(|e| queue_problem(&e))? + { + tracing::info!(domain = %what, revision = written, "changed the egress deny list"); + let behind = behind_hives(state, written, &list).await; + return Ok(Json(view(list, written, behind))); + } + } + Err(queue_problem(&anyhow::anyhow!( + "gave up changing {what} after {MAX_ATTEMPTS} conflicting writes" + ))) +} + +/// Refuse one destination, for every agent or for one. +/// +/// Adding an entry that is already there changes nothing. The entry is +/// refused before any allow rule on every hive that has taken the list. +#[utoipa::path( + post, + path = "/api/egress/denied-domains", + request_body = DeniedDomain, + responses( + (status = 200, description = "the whole list after the add", body = DeniedDomains), + (status = 400, description = "the domain is not a DNS name (an IP address, a \ + wildcard, a single label and a port are refused), the agent is not an \ + identifier or not declared on any hive, or the list is full (problem+json)", + body = String), + (status = 422, description = "an unknown field"), + (status = 503, description = "no swarm queue is configured on this host, or it is \ + not connected (problem+json)", body = String), + (status = 500, description = "the list could not be written (problem+json)", + body = String), + ), + tag = "egress" +)] +pub(crate) async fn add_denied_domain( + State(state): State, + Json(req): Json, +) -> Result, ProblemDetails> { + let entry = checked(&req).map_err(|why| error_problem(StatusCode::BAD_REQUEST, &why))?; + let client = queue(&state).map_err(refused)?; + if let Some(agent) = &entry.agent + && !live(&state).await?.contains(agent) + { + return Err(error_problem( + StatusCode::BAD_REQUEST, + &format!("no hive's wanted state declares an agent {agent:?} that is not destroyed"), + )); + } + apply(&state, &client, &entry.domain, |list, revision| { + added(list, &entry, revision) + }) + .await +} + +/// Stop refusing one destination for the scope it was added with. +/// +/// Removing an entry that is not there changes nothing. A swarm-wide entry is +/// removed only by naming it without an agent: removing an agent's entry for +/// the same name leaves it refused for everyone. +#[utoipa::path( + delete, + path = "/api/egress/denied-domains", + request_body = DeniedDomain, + responses( + (status = 200, description = "the whole list after the removal", body = DeniedDomains), + (status = 400, description = "the domain is not a DNS name, or the agent is not an \ + identifier (problem+json)", body = String), + (status = 422, description = "an unknown field"), + (status = 503, description = "no swarm queue is configured on this host, or it is \ + not connected (problem+json)", body = String), + (status = 500, description = "the list could not be written (problem+json)", + body = String), + ), + tag = "egress" +)] +pub(crate) async fn remove_denied_domain( + State(state): State, + Json(req): Json, +) -> Result, ProblemDetails> { + let entry = checked(&req).map_err(|why| error_problem(StatusCode::BAD_REQUEST, &why))?; + let client = queue(&state).map_err(refused)?; + apply(&state, &client, &entry.domain, |list, revision| { + Ok(removed(list, &entry, revision)) + }) + .await +} + +/// Drop every entry of a destroyed agent: a name is free again once its agent +/// is destroyed, and the next agent of that name would inherit what it left. A +/// list that was never written stays unwritten. A failure is logged and does +/// not stop the destroy, as in `egress_allow::clear_on_destroy`. +pub(crate) async fn clear_on_destroy(state: &AppState, hive: &str, agent: &str) { + let cleared = async { + let client = queue(state).map_err(|(_, why)| anyhow::anyhow!(why))?; + let Some(store) = egress_allow::open(&client).await? else { + return Ok(()); + }; + for _ in 0..MAX_ATTEMPTS { + let (mut list, revision, _) = read_or_default::(&store, DENY_KEY).await?; + let before = list.entries.len(); + list.entries.retain(|e| e.agent.as_deref() != Some(agent)); + if list.entries.len() == before { + return Ok(()); + } + if write_revision(&store, DENY_KEY, &list, revision) + .await? + .is_some() + { + return Ok(()); + } + } + anyhow::bail!("gave up after {MAX_ATTEMPTS} conflicting writes") + } + .await; + match cleared { + Ok(()) => tracing::info!(hive, agent, "destroyed agent's denied domains cleared"), + Err(e) => tracing::error!( + hive, + agent, + error = %format!("{e:#}"), + "clearing this agent's denied domains FAILED; it was destroyed anyway, so a new \ + agent of the same name inherits them. Declaring the agent destroyed again re-runs this." + ), + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::agent_limits::tests::{declared, state}; + use crate::test_nats::TestNats; + use swarm_wire::hive_status::EGRESS_DENY_APPLIED; + + fn entry(domain: &str, agent: Option<&str>) -> DeniedDomain { + DeniedDomain { + domain: domain.to_owned(), + agent: agent.map(str::to_owned), + } + } + + async fn add( + state: &AppState, + domain: &str, + agent: Option<&str>, + ) -> Result { + add_denied_domain(State(state.clone()), Json(entry(domain, agent))) + .await + .map(|j| j.0) + } + + async fn remove( + state: &AppState, + domain: &str, + agent: Option<&str>, + ) -> Result { + remove_denied_domain(State(state.clone()), Json(entry(domain, agent))) + .await + .map(|j| j.0) + } + + async fn read(state: &AppState) -> DeniedDomains { + get_denied_domains(State(state.clone())) + .await + .expect("the list is read") + .0 + } + + async fn listed(state: &AppState) -> Vec<(String, Option)> { + read(state) + .await + .entries + .into_iter() + .map(|e| (e.domain, e.agent)) + .collect() + } + + /// The raw value a hive reads under `key`, or `None` for no key. + async fn published(client: &async_nats::Client, key: &str) -> Option { + egress_allow::open_or_create(client) + .await + .expect("bucket") + .get(key) + .await + .expect("read") + .map(|v| String::from_utf8(v.to_vec()).expect("utf-8")) + } + + fn row(name: &str, freshness: Freshness, applied: Option) -> HiveStatus { + HiveStatus { + name: name.to_owned(), + domain: None, + dashboard: None, + freshness, + last_seen_unix: None, + age_seconds: None, + snapshot: applied.map(|r| serde_json::json!({ EGRESS_DENY_APPLIED: r })), + } + } + + #[test] + fn a_name_is_stored_normalised_and_an_ip_literal_is_refused() { + let stored = checked(&entry(" Evil.COM ", Some("atlas"))).expect("a name"); + assert_eq!(stored.domain, "evil.com"); + assert_eq!(stored.agent.as_deref(), Some("atlas")); + let dotted = checked(&entry(".evil.com", None)).expect("dot form"); + assert_eq!(dotted.domain, ".evil.com"); + + for bad in [ + "1.2.3.4", + "[::1]", + "*.example.com", + "localhost", + "example.com:443", + "example.com\nhttp_access allow all", + ] { + assert!(checked(&entry(bad, None)).is_err(), "{bad:?} was accepted"); + } + for agent in ["", "At Las", "a/b", "../argus"] { + assert!( + checked(&entry("evil.com", Some(agent))).is_err(), + "agent {agent:?} was accepted" + ); + } + } + + /// The base list is no reason to refuse a deny: denying a name that is + /// otherwise always allowed is what the list is for. + #[tokio::test] + async fn a_name_the_base_list_allows_can_be_denied() { + let nats = TestNats::start(); + let client = nats.client().await; + let state = state(&client); + // Control: the allow list's own check refuses this name. + assert!( + crate::egress_allow::checked( + &swarm_controller_api::egress::AllowedDomain { + domain: "api.anthropic.com".to_owned(), + agent: None, + derived: false, + }, + &["api.anthropic.com".to_owned()], + ) + .is_err() + ); + add(&state, "api.anthropic.com", None) + .await + .expect("denied despite the base list"); + } + + /// Through the real handlers and a real `nats-server`: an entry lands under + /// the deny key a hive reads and never under the allow key, the two scopes + /// of one name are two entries, a repeat writes nothing, and removing an + /// agent's entry leaves the swarm-wide one in force. + /// + /// The published bytes are pinned literally because hive-c0re's + /// `workers::squid_deny` tests decode this same string. + #[tokio::test] + async fn the_route_writes_the_list_a_hive_reads_and_a_scoped_removal_never_lifts_the_swarm_wide_entry() + { + let nats = TestNats::start(); + let client = nats.client().await; + let state = declared(&client).await; + + add(&state, "evil.com", None).await.expect("swarm-wide"); + add(&state, "bad.io", Some("atlas")).await.expect("atlas's"); + add(&state, "evil.com", Some("atlas")) + .await + .expect("the same name for atlas alone"); + let before = published(&client, DENY_KEY).await; + add(&state, "EVIL.com", None).await.expect("a repeat"); + assert_eq!( + published(&client, DENY_KEY).await, + before, + "a repeat writes nothing" + ); + + assert_eq!( + published(&client, DENY_KEY).await.as_deref(), + Some( + r#"{"entries":[{"domain":"bad.io","agent":"atlas"},{"domain":"evil.com"},{"domain":"evil.com","agent":"atlas"}]}"# + ) + ); + assert_eq!( + published(&client, swarm_wire::egress_allow::KEY).await, + None, + "a deny is not an allow entry" + ); + + remove(&state, "evil.com", Some("atlas")) + .await + .expect("removes atlas's"); + assert_eq!( + listed(&state).await, + [ + ("bad.io".to_owned(), Some("atlas".to_owned())), + ("evil.com".to_owned(), None), + ], + "the swarm-wide deny stays after atlas's is removed" + ); + let before = published(&client, DENY_KEY).await; + remove(&state, "evil.com", Some("atlas")) + .await + .expect("already gone"); + assert_eq!(published(&client, DENY_KEY).await, before); + + remove(&state, "evil.com", None).await.expect("swarm-wide"); + remove(&state, "bad.io", Some("atlas")) + .await + .expect("atlas's"); + assert_eq!( + published(&client, DENY_KEY).await.as_deref(), + Some(r#"{"entries":[]}"#), + "the last removal leaves an empty list" + ); + } + + /// An allow-list write leaves the deny list alone and the other way round. + #[tokio::test] + async fn the_allow_list_and_the_deny_list_are_separate_values() { + let nats = TestNats::start(); + let client = nats.client().await; + let state = declared(&client).await; + + add(&state, "evil.com", None).await.expect("a deny"); + let allowed = crate::egress_allow::add_allowed_domain( + State(state.clone()), + Json(swarm_controller_api::egress::AllowedDomain { + domain: "github.com".to_owned(), + agent: None, + derived: false, + }), + ) + .await + .expect("an allow"); + assert_eq!(allowed.0.entries.len(), 1); + assert_eq!( + published(&client, DENY_KEY).await.as_deref(), + Some(r#"{"entries":[{"domain":"evil.com"}]}"#) + ); + assert_eq!( + published(&client, swarm_wire::egress_allow::KEY) + .await + .as_deref(), + Some(r#"{"entries":[{"domain":"github.com"}]}"#) + ); + } + + #[tokio::test] + async fn a_read_before_any_entry_answers_empty_and_creates_no_bucket() { + let nats = TestNats::start(); + let client = nats.client().await; + let state = state(&client); + + let read = read(&state).await; + assert!(read.entries.is_empty() && read.revision == 0 && read.behind.is_empty()); + assert!( + egress_allow::open(&client) + .await + .expect("the queue answers") + .is_none() + ); + } + + #[tokio::test] + async fn an_agent_no_hive_declares_is_refused_and_nothing_is_written() { + let nats = TestNats::start(); + let client = nats.client().await; + let state = declared(&client).await; + + let problem = add(&state, "evil.com", Some("iris")) + .await + .expect_err("no hive declares iris"); + assert_eq!(problem.status, Some(StatusCode::BAD_REQUEST), "{problem:?}"); + assert_eq!(published(&client, DENY_KEY).await, None); + add(&state, "evil.com", Some("atlas")) + .await + .expect("control: a declared agent is accepted"); + } + + #[test] + fn a_full_list_takes_no_new_entry_but_a_present_one_is_not_a_new_one() { + let full = DenyList { + entries: (0..MAX_ENTRIES) + .map(|i| DenyEntry { + domain: format!("host{i}.example.com"), + agent: None, + }) + .collect(), + }; + let new = DenyEntry { + domain: "evil.com".to_owned(), + agent: None, + }; + let why = added(full.clone(), &new, 1).err().expect("a full list"); + assert!(why.contains("1000"), "{why}"); + let present = full.entries[0].clone(); + assert!(matches!( + added(full.clone(), &present, 1), + Ok(Changed::Same(..)) + )); + let mut room = full; + room.entries.pop(); + let Ok(Changed::New(list)) = added(room, &new, 1) else { + panic!("a list with room takes the entry"); + }; + assert_eq!(list.entries.len(), MAX_ENTRIES); + } + + /// A hive is behind unless it reports the list's revision; one that reports + /// nothing is behind only while the list asks something of it. + #[test] + fn a_hive_is_behind_unless_it_reports_the_revision_it_should_have() { + let rows = [ + row("current", Freshness::Fresh, Some(7)), + row("older", Freshness::Fresh, Some(5)), + row("silent-old", Freshness::Fresh, None), + row("never", Freshness::NeverReported, None), + row("stale-current", Freshness::Stale, Some(7)), + row("stranger", Freshness::Unknown, None), + ]; + assert_eq!(behind(&rows, 7, false), ["older", "silent-old", "never"]); + // An empty list asks nothing of a hive that reports nothing, but a hive + // that reports a different revision still holds names the list dropped. + assert_eq!(behind(&rows, 7, true), ["older"]); + assert_eq!( + behind(&rows, 0, true), + ["current", "older", "stale-current"] + ); + } + + /// Through the status bucket: after an add every hive is behind until it + /// publishes the revision the add answered. + #[tokio::test] + async fn a_hive_leaves_behind_when_its_snapshot_carries_the_revision() { + let nats = TestNats::start(); + let client = nats.client().await; + let state = declared(&client).await; + + let added = add(&state, "evil.com", None).await.expect("a deny"); + assert!(added.revision > 0); + assert_eq!(added.behind, ["pr1ma"], "it has not reported"); + + let status = swarm_queue_client::status::open_or_create(&client) + .await + .expect("status bucket"); + let publish = |applied: u64| { + let body = serde_json::json!({ + "status": "ok", "warnings": [], EGRESS_DENY_APPLIED: applied + }); + let status = status.clone(); + async move { + status + .put("pr1ma", body.to_string().into()) + .await + .expect("publish"); + } + }; + publish(added.revision - 1).await; + assert_eq!(read(&state).await.behind, ["pr1ma"], "an older revision"); + publish(added.revision).await; + let now = read(&state).await; + assert_eq!(now.revision, added.revision); + assert!(now.behind.is_empty(), "{:?}", now.behind); + } + + #[tokio::test] + async fn destroying_an_agent_drops_its_denies_and_keeps_the_rest() { + let nats = TestNats::start(); + let client = nats.client().await; + let state = declared(&client).await; + add(&state, "evil.com", None).await.expect("swarm-wide"); + add(&state, "bad.io", Some("atlas")).await.expect("atlas's"); + + clear_on_destroy(&state, "pr1ma", "atlas").await; + assert_eq!(listed(&state).await, [("evil.com".to_owned(), None)]); + } + + async fn call(app: &axum::Router, method: &str, body: &'static str, path: &str) -> StatusCode { + use tower::ServiceExt as _; + app.clone() + .oneshot( + axum::http::Request::builder() + .method(method) + .uri(path) + .header(axum::http::header::CONTENT_TYPE, "application/json") + .body(axum::body::Body::from(body)) + .expect("request"), + ) + .await + .expect("router answers") + .status() + } + + /// Through the real router: an IP literal or a wildcard is a 400 and writes + /// nothing, an unknown field (a `derived` flag included) a 422, and the + /// well-formed body after them is the control that the route reaches the + /// deny key. + #[tokio::test] + async fn the_router_refuses_bad_names_and_a_derived_flag_before_the_queue() { + let nats = TestNats::start(); + let client = nats.client().await; + let app = crate::startup::build_app(declared(&client).await); + let path = "/api/egress/denied-domains"; + + for body in [ + r#"{"domain":"1.2.3.4"}"#, + r#"{"domain":"*.example.com"}"#, + r#"{"domain":""}"#, + ] { + assert_eq!( + call(&app, "POST", body, path).await, + StatusCode::BAD_REQUEST, + "{body}" + ); + } + for body in [ + r#"{"domain":"evil.com","derived":true}"#, + r#"{"domain":"evil.com","hive":"pr1ma"}"#, + ] { + assert_eq!( + call(&app, "POST", body, path).await, + StatusCode::UNPROCESSABLE_ENTITY, + "{body}" + ); + } + assert_eq!(published(&client, DENY_KEY).await, None); + + assert_eq!( + call(&app, "POST", r#"{"domain":"evil.com"}"#, path).await, + StatusCode::OK + ); + assert_eq!( + published(&client, DENY_KEY).await.as_deref(), + Some(r#"{"entries":[{"domain":"evil.com"}]}"#) + ); + assert_eq!( + call(&app, "DELETE", r#"{"domain":"evil.com"}"#, path).await, + StatusCode::OK + ); + } +} diff --git a/swarm-controller/src/main.rs b/swarm-controller/src/main.rs index a24c5873..e97cb079 100644 --- a/swarm-controller/src/main.rs +++ b/swarm-controller/src/main.rs @@ -49,6 +49,7 @@ mod claude_login; mod config_drift; mod config_pr; mod egress_allow; +mod egress_deny; mod forge; mod forge_account; mod forge_events; diff --git a/swarm-controller/src/routes/agents.rs b/swarm-controller/src/routes/agents.rs index 4c8baf01..250602ca 100644 --- a/swarm-controller/src/routes/agents.rs +++ b/swarm-controller/src/routes/agents.rs @@ -19,7 +19,7 @@ use crate::routes::hives::HiveEntry; use crate::routes::{AppState, StatusUnavailable, error_problem, wanted_error_status}; use crate::{ agent_archetype, agent_identity, agent_limits, agent_permissions, agent_status, - agents_md_initial, auth, egress_allow, inference, schedules, wanted, + agents_md_initial, auth, egress_allow, egress_deny, inference, schedules, wanted, }; /// The swarm's agent roster — every agent the swarm holds an identity for. @@ -217,6 +217,7 @@ pub(crate) async fn set_agent_state( agent_archetype::clear_on_destroy(&state, &hive, &agent).await; agents_md_initial::clear_on_destroy(&state, &hive, &agent).await; egress_allow::clear_on_destroy(&state, &hive, &agent).await; + egress_deny::clear_on_destroy(&state, &hive, &agent).await; } Ok(Json(render(&declaration))) } diff --git a/swarm-controller/src/startup.rs b/swarm-controller/src/startup.rs index a0a7259c..8da7568c 100644 --- a/swarm-controller/src/startup.rs +++ b/swarm-controller/src/startup.rs @@ -31,9 +31,10 @@ use crate::{ agent_control, agent_custom, agent_icon, agent_input, agent_limits, agent_login, agent_permissions, agent_renewal, agent_state_stream, agent_status, agents_md, alerts, archetypes, auth, backend_env, builder, claude_login, config_drift, config_pr, egress_allow, - forge, forge_account, github_account, gitlab_account, inference, issue_report, linked_accounts, - matrix_account, message_stream, notify_flows, notify_router_api, pulls, queue_identity, - review_classifier, schedules, status, subagent_term, term_stream, wanted, webhook, + egress_deny, forge, forge_account, github_account, gitlab_account, inference, issue_report, + linked_accounts, matrix_account, message_stream, notify_flows, notify_router_api, pulls, + queue_identity, review_classifier, schedules, status, subagent_term, term_stream, wanted, + webhook, }; /// Where the daemon binds, overridable via `SWARM_CONTROLLER_SOCKET`. @@ -649,6 +650,11 @@ pub(crate) fn build_app(state: AppState) -> axum::Router { egress_allow::add_allowed_domain, egress_allow::remove_allowed_domain )) + .routes(routes!( + egress_deny::get_denied_domains, + egress_deny::add_denied_domain, + egress_deny::remove_denied_domain + )) .routes(routes!(archetypes::get_archetypes)) .routes(routes!(config_prs::get_config_prs)) .routes(routes!(agents::create_agent)) diff --git a/swarm-nats-auth/src/policy.rs b/swarm-nats-auth/src/policy.rs index 9acdc172..0d5db780 100644 --- a/swarm-nats-auth/src/policy.rs +++ b/swarm-nats-auth/src/policy.rs @@ -613,6 +613,9 @@ impl Policy { // The egress allow-list (`swarm_wire::egress_allow`): one key, read with // one direct get, so exactly that subject and no other key of the bucket. subjects.push(swarm_wire::egress_allow::direct_get_subject()); + // The deny list is a second key of that bucket, read the same way; the + // allow-list grant does not cover it, since it names one key. + subjects.push(swarm_wire::egress_allow::deny_direct_get_subject()); // The archetype an agent was created from // (`swarm_queue_client::agent_archetype`), read the same way and every // key for the same reason. @@ -984,6 +987,48 @@ mod tests { } } + /// A hive reads the deny list with its own grant, not the allow-list's, and + /// can change none of it. An agent's token reaches neither key, so nothing an + /// agent holds can write the list a hive enforces. + #[test] + fn a_hive_reads_the_deny_list_and_no_agent_reaches_it() { + use swarm_wire::egress_allow::{ + STREAM, deny_direct_get_subject, deny_key_subject, direct_get_subject, key_subject, + }; + + let p = policy() + .permissions("hive-alpha") + .expect("a hive is admitted"); + let may = |subject: &str| p.publish.iter().any(|g| covers(g, subject)); + assert!(may(&deny_direct_get_subject()), "cannot read the deny list"); + for forbidden in [ + deny_key_subject(), + format!("$JS.API.STREAM.PURGE.{STREAM}"), + format!("$JS.API.STREAM.MSG.DELETE.{STREAM}"), + ] { + assert!(!may(&forbidden), "hive alpha may publish {forbidden}"); + } + + let agent = policy() + .with_agent_token_subjects(vec!["$SWARM.term.{agent}".to_owned()]) + .expect("a per-agent template is valid") + .agent_token_permissions("atlas") + .expect("configured"); + let agent_may = |subject: &str| agent.publish.iter().any(|g| covers(g, subject)); + for reached in [ + deny_key_subject(), + deny_direct_get_subject(), + key_subject(), + direct_get_subject(), + ] { + assert!(!agent_may(&reached), "an agent may publish {reached}"); + } + // Control: the lookup finds a subject the agent is granted. + assert!(agent_may( + &swarm_queue_client::operator_profiles::direct_get_subject() + )); + } + /// A hive reads any agent's archetype and can change none of them, and the /// grant reaches no other bucket. #[test] diff --git a/swarm-queue-client/src/egress_allow.rs b/swarm-queue-client/src/egress_allow.rs index 90b280f0..77505409 100644 --- a/swarm-queue-client/src/egress_allow.rs +++ b/swarm-queue-client/src/egress_allow.rs @@ -56,7 +56,22 @@ pub enum Read { NotSet, } -/// Read the current list with one direct get. +/// What reading the deny list found. +#[cfg(feature = "egress-allow")] +#[derive(Debug)] +pub enum DenyRead { + /// The controller has written a deny list, which may be empty, at this + /// revision of its key: the stream sequence of the write, which is what the + /// controller's own read of the key reports as its revision. + Current { + list: swarm_wire::egress_allow::DenyList, + revision: u64, + }, + /// There is no deny list to read, for the reasons [`Read::NotSet`] gives. + NotSet, +} + +/// Read the current allow-list with one direct get. /// /// # Errors /// @@ -66,15 +81,50 @@ pub enum Read { pub async fn read( client: &async_nats::Client, ) -> Result> { + let Some(message) = last_value(client, &swarm_wire::egress_allow::key_subject()).await? else { + return Ok(Read::NotSet); + }; + Ok(Read::Current(serde_json::from_slice(&message.payload)?)) +} + +/// Read the current deny list with one direct get. +/// +/// # Errors +/// +/// As [`read`]. +#[cfg(feature = "egress-allow")] +pub async fn read_deny( + client: &async_nats::Client, +) -> Result> { + let Some(message) = last_value(client, &swarm_wire::egress_allow::deny_key_subject()).await? + else { + return Ok(DenyRead::NotSet); + }; + Ok(DenyRead::Current { + list: serde_json::from_slice(&message.payload)?, + revision: message.sequence, + }) +} + +/// The last message under `subject` of the bucket's stream, or `None` when +/// there is none or it is a KV delete marker. +#[cfg(feature = "egress-allow")] +async fn last_value( + client: &async_nats::Client, + subject: &str, +) -> Result< + Option, + Box, +> { use async_nats::jetstream::stream::DirectGetErrorKind; - use swarm_wire::egress_allow::{STREAM, key_subject}; + use swarm_wire::egress_allow::STREAM; let js = async_nats::jetstream::new(client.clone()); let stream = js.get_stream_no_info(STREAM).await?; - let message = match stream.direct_get_last_for_subject(key_subject()).await { + let message = match stream.direct_get_last_for_subject(subject).await { Ok(message) => message, - Err(e) if matches!(e.kind(), DirectGetErrorKind::NotFound) => return Ok(Read::NotSet), - Err(e) if is_no_responders(&e) => return Ok(Read::NotSet), + Err(e) if matches!(e.kind(), DirectGetErrorKind::NotFound) => return Ok(None), + Err(e) if is_no_responders(&e) => return Ok(None), Err(e) => return Err(e.into()), }; // A KV delete or purge is stored as an empty marker carrying this header. @@ -82,10 +132,7 @@ pub async fn read( .headers .get("KV-Operation") .is_some_and(|op| matches!(op.as_str(), "DEL" | "PURGE")); - if removed { - return Ok(Read::NotSet); - } - Ok(Read::Current(serde_json::from_slice(&message.payload)?)) + Ok((!removed).then_some(message)) } #[cfg(feature = "egress-allow")] @@ -96,3 +143,75 @@ fn is_no_responders(e: &async_nats::jetstream::stream::DirectGetError) -> bool { if status == async_nats::StatusCode::NO_RESPONDERS ) } + +#[cfg(all(test, feature = "kv", feature = "egress-allow"))] +mod tests { + use swarm_wire::egress_allow::{DENY_KEY, DenyEntry, DenyList, KEY}; + + use super::{DenyRead, Read, open_or_create, read, read_deny}; + use crate::test_nats::TestNats; + + /// The deny list is read from its own key with the revision of the write, + /// the allow-list's key is not mistaken for it, and a deleted or never + /// written key reads as unset. + #[tokio::test] + async fn the_deny_list_is_read_from_its_own_key_with_the_revision_it_was_written_at() { + let nats = TestNats::start(); + let client = nats.client().await; + + assert!(matches!( + read_deny(&client).await.expect("no bucket"), + DenyRead::NotSet + )); + let store = open_or_create(&client).await.expect("bucket"); + assert!(matches!( + read_deny(&client).await.expect("no key"), + DenyRead::NotSet + )); + + let list = DenyList { + entries: vec![DenyEntry { + domain: ".evil.example".to_owned(), + agent: None, + }], + }; + store + .put( + KEY, + br#"{"entries":[{"domain":"allowed.example"}]}"#.to_vec().into(), + ) + .await + .expect("an allow-list write"); + assert!( + matches!( + read_deny(&client).await.expect("still unset"), + DenyRead::NotSet + ), + "the allow-list's key is not the deny list" + ); + let revision = store + .put(DENY_KEY, serde_json::to_vec(&list).expect("encodes").into()) + .await + .expect("a deny-list write"); + match read_deny(&client).await.expect("read") { + DenyRead::Current { + list: got, + revision: at, + } => { + assert_eq!(got, list); + assert_eq!(at, revision, "the revision the controller's write answered"); + } + DenyRead::NotSet => panic!("the list was written"), + } + assert!( + matches!(read(&client).await.expect("allow read"), Read::Current(l) if l.entries.len() == 1), + "control: the allow-list read still finds its own key" + ); + + store.delete(DENY_KEY).await.expect("deleted"); + assert!(matches!( + read_deny(&client).await.expect("deleted"), + DenyRead::NotSet + )); + } +} diff --git a/swarm-queue-client/src/lib.rs b/swarm-queue-client/src/lib.rs index e6a3dd18..61101db4 100644 --- a/swarm-queue-client/src/lib.rs +++ b/swarm-queue-client/src/lib.rs @@ -487,7 +487,14 @@ pub mod forge_events; #[cfg(feature = "router-todo")] pub mod router_todo; -#[cfg(all(test, any(feature = "forge-events", feature = "router-todo")))] +#[cfg(all( + test, + any( + feature = "forge-events", + feature = "router-todo", + all(feature = "kv", feature = "egress-allow") + ) +))] mod test_nats; /// Only the fields this needs; authelia returns several. diff --git a/swarm-wire/src/egress_allow.rs b/swarm-wire/src/egress_allow.rs index 1f05035f..10a9c71d 100644 --- a/swarm-wire/src/egress_allow.rs +++ b/swarm-wire/src/egress_allow.rs @@ -1,5 +1,6 @@ //! The egress allow-list KV bucket: one key holding the destination names the -//! swarm lets agents reach through their hive's squid. +//! swarm lets agents reach through their hive's squid, and a second holding the +//! names it refuses. //! //! **swarm-controller is the only writer**, and creates the bucket on its //! first write. Each [`Entry`] is a domain with a scope: every agent in the @@ -10,6 +11,11 @@ //! covers the bucket read-only, and renders the entries for the agents it runs. //! The names go into squid's configuration, so [`normalize_domain`] is the one //! gate both ends use: the controller at write, the hive again at render. +//! +//! The deny list is its own key ([`DENY_KEY`]) of the same bucket, with its own +//! type ([`DenyList`]): a reader that predates it never reads the key, and a +//! controller that predates it never rewrites it, so neither side can drop the +//! other's entries by writing its own value back. /// The KV bucket the list is published into. pub const BUCKET: &str = "egress-allow"; @@ -32,6 +38,21 @@ pub fn direct_get_subject() -> String { format!("$JS.API.DIRECT.GET.{STREAM}.{}", key_subject()) } +/// The key the deny list is stored under, beside [`KEY`]. +pub const DENY_KEY: &str = "deny"; + +/// The subject the deny list is stored under. +#[must_use] +pub fn deny_key_subject() -> String { + format!("$KV.{BUCKET}.{DENY_KEY}") +} + +/// The request subject that reads the current deny list. +#[must_use] +pub fn deny_direct_get_subject() -> String { + format!("$JS.API.DIRECT.GET.{STREAM}.{}", deny_key_subject()) +} + /// The most entries a list holds, which keeps the value far below the queue's /// message size and the rendered squid configuration small. pub const MAX_ENTRIES: usize = 1000; @@ -89,6 +110,29 @@ impl Entry { } } +/// One refused destination and who it is refused for. +/// +/// A distinct type from [`Entry`]: a deny is never derived from an agent's +/// environment or a linked account, so it carries neither field and nothing +/// but an operator's action can put one in the list. +#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, serde::Serialize, serde::Deserialize)] +pub struct DenyEntry { + /// A name as [`normalize_domain`] returns it. + pub domain: String, + /// The agent this entry is for; absent means every agent in the swarm. An + /// agent's entry adds to the swarm-wide ones and never lifts one. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub agent: Option, +} + +/// The value under [`BUCKET`]/[`DENY_KEY`]. No `deny_unknown_fields`, as for +/// [`AllowList`]. +#[derive(Debug, Clone, Default, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +pub struct DenyList { + #[serde(default)] + pub entries: Vec, +} + /// The value under [`BUCKET`]/[`KEY`]. No `deny_unknown_fields`: a field a /// newer controller adds must not cost an older hive the entries it reads. #[derive(Debug, Clone, Default, PartialEq, Eq, serde::Serialize, serde::Deserialize)] @@ -180,6 +224,39 @@ mod tests { ); } + #[test] + fn the_deny_key_is_its_own_subject_of_the_same_bucket() { + assert_eq!(deny_key_subject(), "$KV.egress-allow.deny"); + assert_eq!( + deny_direct_get_subject(), + "$JS.API.DIRECT.GET.KV_egress-allow.$KV.egress-allow.deny" + ); + assert_ne!(deny_key_subject(), key_subject()); + } + + #[test] + fn a_deny_entry_round_trips_and_tolerates_fields_it_does_not_know() { + let list = DenyList { + entries: vec![ + DenyEntry { + domain: ".evil.example".to_owned(), + agent: None, + }, + DenyEntry { + domain: "bad.io".to_owned(), + agent: Some("atlas".to_owned()), + }, + ], + }; + let wire = + r#"{"entries":[{"domain":".evil.example"},{"domain":"bad.io","agent":"atlas"}]}"#; + assert_eq!(serde_json::to_string(&list).unwrap(), wire); + assert_eq!(serde_json::from_str::(wire).unwrap(), list); + let later: DenyList = + serde_json::from_str(r#"{"entries":[{"domain":"a.io","later":1}],"x":1}"#).unwrap(); + assert_eq!(later.entries.len(), 1); + } + #[test] fn a_dns_name_is_kept_lower_cased_with_one_optional_leading_dot() { for (raw, stored) in [ diff --git a/swarm-wire/src/hive_status.rs b/swarm-wire/src/hive_status.rs index 8da85459..5158483c 100644 --- a/swarm-wire/src/hive_status.rs +++ b/swarm-wire/src/hive_status.rs @@ -18,6 +18,20 @@ pub const TOLERANT_PERMISSIONS: &str = "tolerant-permissions"; /// into a custom one only on a hive that advertises it. pub const NO_CONFIG_CHECKOUT: &str = "no-config-checkout"; +/// The snapshot field holding the revision of the egress deny list +/// (`crate::egress_allow::DENY_KEY`) the hive last rendered into its squid +/// include, `0` when the swarm has no deny list. A hive that predates the +/// field publishes none. +pub const EGRESS_DENY_APPLIED: &str = "egress_deny_applied"; + +/// The deny-list revision a stored snapshot says its hive rendered: `None` for +/// a snapshot without the field (an older hive, or one that has not read the +/// list yet) or with one that is not a number. +#[must_use] +pub fn egress_deny_applied(snapshot: &serde_json::Value) -> Option { + snapshot.get(EGRESS_DENY_APPLIED)?.as_u64() +} + /// The part of a hive's status snapshot this module describes. #[derive(Debug, Clone, Default, PartialEq, Eq, serde::Serialize, serde::Deserialize)] pub struct Capabilities { @@ -66,6 +80,21 @@ mod tests { assert!(!Capabilities::of_snapshot(&new).has("other")); } + #[test] + fn the_applied_deny_revision_is_read_only_when_it_is_a_number() { + let at = |v: serde_json::Value| { + egress_deny_applied(&serde_json::json!({ EGRESS_DENY_APPLIED: v })) + }; + assert_eq!(at(serde_json::json!(7)), Some(7)); + assert_eq!(at(serde_json::json!(0)), Some(0)); + assert_eq!(at(serde_json::json!("7")), None); + assert_eq!( + egress_deny_applied(&serde_json::json!({"status": "ok"})), + None + ); + assert_eq!(egress_deny_applied(&serde_json::json!(null)), None); + } + /// A snapshot that is not an object, or whose field is mistyped, grants /// nothing rather than failing the read. #[test]