Watch
0
0
Fork
You've already forked hyperhive
0

Compare commits

...
19 changed files with 1208 additions and 96 deletions

View file

@ -295,10 +295,9 @@ host. It never decrypts TLS, so it holds no signing key. It caches nothing.
(<https://code.claude.com/docs/en/network-config>): `api.anthropic.com` for
the model API and the usage probe (`hive-agent/src/claude_usage_watch.rs`),
`claude.ai` for the account login, and `platform.claude.com` for the OAuth
token exchange and refresh. An agent whose `ANTHROPIC_BASE_URL` or ACP
provider points elsewhere needs that host added to the option: the value
sits in the agent's own environment file, which the host doesn't read. The
list omits the plugin marketplace clone from `github.com`, since a failed add
token exchange and refresh. An agent whose `ANTHROPIC_BASE_URL` points
elsewhere needs no entry here: the swarm controller adds that host to the
agent's own swarm entries (see _Swarm allow-list_ below). The list omits the plugin marketplace clone from `github.com`, since a failed add
isn't fatal. The rules below still refuse an entry that resolves to a
private address.
- **Allow-list.** `services.hyperhive.squid.allowedDomains` defaults to empty,
@ -308,8 +307,8 @@ host. It never decrypts TLS, so it holds no signing key. It caches nothing.
which the rules below refuse, and agents reach them directly. Each entry
matches that exact name, or with a leading dot the name and its subdomains. A user on the unrestricted list reaches any
other name too, on ports 80 and 443. Everyone else gets a 403 for it.
`allowedDomains` is the hive's own list. Squid uses it until the swarm has a
list of its own (next bullet).
`allowedDomains` is the hive's own list, always allowed whether or not the
swarm has a list of its own (next bullet).
- **Swarm allow-list.** The operator adds domains in the swarm UI
([allowed domains](../swarm/ui.md#allowed-domains)); the swarm controller
keeps them in one value of the `egress-allow` bucket, each for every agent
@ -333,11 +332,30 @@ host. It never decrypts TLS, so it holds no signing key. It caches nothing.
you add the entry, and `hive-c0re` checks each name again and skips one that
fails, or that the base list or the canary covers, or that another entry of
the same list covers. A name that resolves to a private address stays a 403.
The swarm's list replaces the hive's `allowedDomains` as soon as it exists,
even when empty: add the names the hive allowed to the swarm's list
before relying on it. Until the controller has a list (it predates the
The controller also adds entries itself, and only from the operator's save
of an agent's API-key environment in the swarm UI: it takes the host from
`ANTHROPIC_BASE_URL` and from every variable named `*_BASE_URL` (the
address an ACP agent that loads the same file reads), and keeps one
`derived` entry for that agent per host. Each host passes the same check as
an operator's entry, so an IP address, `localhost` and a base-list name get
none. A save or a delete of the environment replaces the agent's derived
entries, and destroying the agent drops all its entries. The agent can
create its own `backend-env` leaf while the leaf is empty, so text found in
the store never creates an entry: when the controller starts, it only
drops derived entries whose exact host the stored environment no longer
names, and logs a host it didn't derive. An environment saved before this
existed gets its entries when the operator saves it once more. An agent on
the hand-placed `backendEnvironmentFile` alone, and the `opencode` preset's
`acp.opencode.provider.baseUrl`, get no derived entry, so add one in the
swarm UI. A `hive-c0re` that predates the field renders a derived entry
like any other.
The swarm's list adds to the hive's `allowedDomains`: `allowedFile` holds
those names plus the swarm-wide entries, so removing an entry never removes a
name the hive lists itself. Until the controller has a list (it predates the
feature, or nobody added an entry) `allowedFile` holds `allowedDomains` and
`agent-allow.conf` is empty. A queue that stops answering for the bucket
`agent-allow.conf` is empty. A `hive-c0re` older than this rule replaces
`allowedDomains` with the swarm's list once the controller has one, so every
hive must run the new `hive-c0re` before the controller's first entry exists. A queue that stops answering for the bucket
leaves a file the swarm wrote untouched, and a failed read leaves both
files untouched. The first line of a file written from the swarm's list is a
`# source: swarm` comment. The queue grant a hive needs to read the list

View file

@ -633,6 +633,13 @@ name the proxy's private-range rules leave. **remove** drops one entry;
**add** opens a dialog with a domain box and a tick for every agent in the
swarm. The tick is off by default, so an entry is for this agent alone.
An entry with the badge **automatic** has no **remove**: the controller keeps
it from the host in the agent's stored API-key environment
(`ANTHROPIC_BASE_URL`, or any `*_BASE_URL` variable), so a restricted agent on
another provider reaches it. Saving or removing the environment changes it, and
the API refuses to add or remove one (`400` and `409`). An agent that only has
the hand-placed environment file gets none; add its provider's host here.
A domain is a DNS name of at least two labels, such as `github.com`. One
leading dot, as in `.githubusercontent.com`, also allows every subdomain; the
dot form is the only wildcard, because exact names can't list a service that
@ -648,9 +655,8 @@ an entry that exists changes nothing.
A saved change isn't instant. Each hive reads the list about every 30
seconds and reloads its proxy, so the name is reachable within a minute. The
proxy's private-range rules still apply: an entry whose name resolves to a
loopback, private or bridge address stays blocked. The first entry makes the
swarm's list replace the names a hive's own `allowedDomains` lists, so add
those names here too; a hive keeps its own list until then. See [the egress
loopback, private or bridge address stays blocked. An entry adds to the names a
hive's own `allowedDomains` lists, which stay allowed. See [the egress
proxy](../networking/network.md#egress-proxy-squid) for what each hive does
when the controller is old or unreachable.

View file

@ -299,6 +299,15 @@ text. An agent whose file has content creates the leaf from it again within 2
minutes; one without a file keeps its copy in `/run` until the container
restarts.
A save also keeps the host of `ANTHROPIC_BASE_URL` (and of every `*_BASE_URL`
variable) as an automatic allowed-domain entry for the agent, so a restricted
agent reaches its provider through the hive's proxy; **Remove** drops it. The
controller derives only from an operator's save, never from text it finds in the
store, so an environment saved before this existed needs one more save. An
agent that has just the file gets no entry, and an operator adds its
provider's host in the swarm UI (see
[allowed domains](../swarm/ui.md#allowed-domains)).
Provision the file once, out of band (never through nix — an API key in
the store is world-readable and travels with the flake closure):

View file

@ -10,12 +10,14 @@
// allows everyone) shows as it arrives, next to the box. The hive reads a
// change within about 30 seconds.
//
// The first entry makes the swarm's list replace the names the hive's own
// configuration allows, so a name the hive allowed and the swarm's list lacks
// stops being allowed once hives read it. An agent that holds the unrestricted
// An entry adds to the names the hive's own configuration allows, which stay
// allowed. An agent that holds the unrestricted
// internet flag reaches any name the proxy's private-range rules leave, so an
// entry matters only to a restricted one.
//
// An entry the controller keeps from the agent's backend environment (`derived`)
// shows as automatic with no remove control; the route refuses to remove it.
//
// Its state belongs to one agent: callers key it by agent name.
import { useEffect, useState } from "preact/hooks";
import { ApiErrorPanel } from "@hive/shared/api-error-panel.js";
@ -117,19 +119,32 @@ export function AllowedDomains({ agent }: { agent: string }) {
{shown.length > 0 ? (
<ul class="ui-agent-limits-list">
{shown.map((e) => (
<li key={`${e.agent ?? "*"}/${e.domain}`}>
<li key={`${e.agent ?? "*"}/${e.domain}/${e.derived === true}`}>
<Badge
label={e.agent === undefined ? "swarm-wide" : agent}
label={
e.derived === true
? "automatic"
: e.agent === undefined
? "swarm-wide"
: agent
}
value={e.domain}
title={
e.derived === true
? "kept from this agent's backend environment; change that variable to change it"
: undefined
}
/>{" "}
<Badge
variant="quiet"
value="remove"
title={`stop allowing ${e.domain} for ${
e.agent === undefined ? "every agent" : agent
}`}
onClick={() => void remove(e)}
/>
{e.derived === true ? null : (
<Badge
variant="quiet"
value="remove"
title={`stop allowing ${e.domain} for ${
e.agent === undefined ? "every agent" : agent
}`}
onClick={() => void remove(e)}
/>
)}
</li>
))}
</ul>
@ -207,9 +222,8 @@ function AddForm({
such as <code>github.com</code>; a leading dot, as in{" "}
<code>.githubusercontent.com</code>, also allows every subdomain. IP
addresses and wildcards are refused. A name that resolves to a private
address stays blocked. The first entry makes the swarm's list replace
the names each hive's own configuration allows. The hive reads a change
within about 30 seconds.
address stays blocked. The entry adds to the names each hive's own
configuration allows. The hive reads a change within about 30 seconds.
</p>
<form class="ui-agent-limits-form" onSubmit={(e) => void save(e)}>
<TextField

View file

@ -7,6 +7,11 @@ export interface AllowedDomain {
domain: string;
/** Absent for an entry that holds for every agent in the swarm. */
agent?: string;
/**
* Kept by the controller from the agent's backend environment: shown as
* automatic, and the route refuses to remove it.
*/
derived?: boolean;
}
export const ALLOWED_DOMAINS_URL = "/api/egress/allowed-domains";
@ -28,10 +33,12 @@ export function entriesFor(
export function allowedSummary(shown: AllowedDomain[]): string {
if (shown.length === 0) return "none added";
const swarmWide = shown.filter((e) => e.agent === undefined).length;
const own = shown.length - swarmWide;
const automatic = shown.filter((e) => e.derived === true).length;
const own = shown.length - swarmWide - automatic;
return [
swarmWide > 0 ? `${swarmWide} swarm-wide` : null,
own > 0 ? `${own} for this agent` : null,
automatic > 0 ? `${automatic} automatic` : null,
]
.filter((part) => part !== null)
.join(", ");

View file

@ -52,3 +52,13 @@ test("the body is swarm-wide or for the agent, with the name trimmed", () => {
agent: "atlas",
});
});
test("a derived entry is counted as automatic, not as the operator's own", () => {
const derived = { domain: "llm.example.com", agent: "atlas", derived: true };
assert.equal(
allowedSummary([...entriesFor(all, "atlas"), derived]),
"1 swarm-wide, 2 for this agent, 1 automatic",
);
assert.equal(allowedSummary([derived]), "1 automatic");
assert.deepEqual(entriesFor([derived], "atlas"), [derived]);
});

View file

@ -15,19 +15,19 @@
//! Squid reads both when it parses its configuration, so the path unit in
//! `nix/host-modules/hive-squid.nix` reloads it when either changes.
//!
//! The swarm's list replaces the names `services.hyperhive.squid.allowedDomains`
//! lists ([`ENV_SEED_FILE`]), which stay the list until the swarm has one. A
//! file rendered from the swarm's list starts with [`SWARM_MARKER`]. The queue
//! not answering for the bucket does not show that the swarm withdrew its list,
//! so a file with the marker is left alone then.
//! The names `services.hyperhive.squid.allowedDomains` lists ([`ENV_SEED_FILE`])
//! are always allowed: the swarm-wide names written are those plus the swarm's,
//! with the per-agent rules on top. Until the swarm has a list the files hold
//! just those names. A file from the swarm's list starts with [`SWARM_MARKER`];
//! the queue not answering for the bucket is no withdrawal, so a file with the
//! marker is left alone then.
//!
//! Every entry is checked again with
//! [`swarm_wire::egress_allow::normalize_domain`], since the names end up in
//! squid's configuration, and skipped when it fails. An entry that
//! [`ENV_ALWAYS_ALLOWED`] or another entry of its list covers is dropped,
//! because squid ignores the narrower of two overlapping names in one ACL.
//!
//! A failed read leaves both files as they are.
//! because squid ignores the narrower of two overlapping names in one ACL. A
//! failed read leaves both files as they are.
use std::path::{Path, PathBuf};
use std::time::Duration;
@ -54,7 +54,7 @@ pub const ENV_ALLOWED_FILE: &str = "HIVE_SQUID_ALLOWED_FILE";
pub const ENV_AGENT_ALLOW_FILE: &str = "HIVE_SQUID_AGENT_ALLOW_FILE";
/// The file of names `services.hyperhive.squid.allowedDomains` lists, one per
/// line, which the files hold until the swarm has a list.
/// line, which every rendering of the swarm-wide names includes.
pub const ENV_SEED_FILE: &str = "HIVE_SQUID_ALLOWED_SEED_FILE";
/// The names, separated by spaces, that squid allows every authenticated user
@ -191,7 +191,7 @@ fn sync_with(
) -> Result<Source> {
let always: Vec<&str> = always.split_whitespace().collect();
let rendered = match read {
Read::Current(list) => render(&list, agents, &always),
Read::Current(list) => render(&list, agents, &always, &seed_names(seed)?),
Read::NotSet => {
if super::squid_users::read_or_empty(files.allowed)?.starts_with(SWARM_MARKER) {
return Ok(Source::Kept);
@ -251,18 +251,27 @@ struct Rendered {
counts: Option<Counts>,
}
/// The files for the seed: its names as they are, and no per-agent rules.
fn seeded(seed: Option<&Path>) -> Result<Rendered> {
let names = match seed {
/// The names in the seed file, trimmed, without blank lines and comments. The
/// file is this hive's own configuration, so its names are not re-checked.
fn seed_names(seed: Option<&Path>) -> Result<Vec<String>> {
let text = match seed {
Some(path) => super::squid_users::read_or_empty(path)?,
None => String::new(),
};
Ok(text
.lines()
.map(str::trim)
.filter(|line| !line.is_empty() && !line.starts_with('#'))
.map(str::to_owned)
.collect())
}
/// The files for the seed: its names as they are, and no per-agent rules.
fn seeded(seed: Option<&Path>) -> Result<Rendered> {
let mut allowed = SEED_MARKER.to_owned();
for line in names.lines().map(str::trim) {
if !line.is_empty() && !line.starts_with('#') {
allowed.push_str(line);
allowed.push('\n');
}
for name in seed_names(seed)? {
allowed.push_str(&name);
allowed.push('\n');
}
Ok(Rendered {
allowed,
@ -283,8 +292,9 @@ fn reduced(mut names: Vec<String>) -> Vec<String> {
names
}
/// The files for `list`, for the agents this hive runs.
fn render(list: &AllowList, agents: &[String], always: &[&str]) -> Rendered {
/// The files for `list` and the seed's `own` names, for the agents this hive
/// runs.
fn render(list: &AllowList, agents: &[String], always: &[&str], own: &[String]) -> Rendered {
let mut skipped = Vec::new();
let mut swarm_wide = Vec::new();
let mut per_agent: std::collections::BTreeMap<&str, Vec<String>> =
@ -305,6 +315,7 @@ fn render(list: &AllowList, agents: &[String], always: &[&str]) -> Rendered {
}
}
}
swarm_wide.extend(own.iter().map(|name| name.to_ascii_lowercase()));
swarm_wide.retain(|d| !covered_by(d, always));
let swarm_wide = reduced(swarm_wide);
@ -349,6 +360,7 @@ mod tests {
Entry {
domain: domain.to_owned(),
agent: agent.map(str::to_owned),
derived: false,
}
}
@ -365,7 +377,7 @@ mod tests {
const ALWAYS: &[&str] = &["api.anthropic.com", ".claude.ai", "cache.nixos.org"];
fn rendered(entries: &[(&str, Option<&str>)], mine: &[&str]) -> Rendered {
render(&list(entries), &agents(mine), ALWAYS)
render(&list(entries), &agents(mine), ALWAYS, &[])
}
#[test]
@ -463,6 +475,7 @@ mod tests {
&list(&[("api.anthropic.com", None), ("a.claude.ai", None)]),
&agents(&["atlas"]),
&[],
&[],
);
assert_eq!(
bare.allowed,
@ -544,18 +557,74 @@ mod tests {
);
}
/// The seed is the hive's own additive list: a swarm list, empty or not,
/// renders the seed's names with it.
#[test]
fn a_swarm_list_replaces_the_seed_even_when_empty() {
fn a_swarm_list_adds_to_the_seed_even_when_empty() {
let d = dir("forge.example\n");
sync(&d, Read::NotSet).expect("seeded");
assert!(text(&d.allowed()).contains("forge.example"), "control");
let empty = Read::Current(AllowList::default());
assert_eq!(sync(&d, empty).expect("renders"), Source::Swarm);
assert_eq!(text(&d.allowed()), SWARM_MARKER);
let named = Read::Current(list(&[("github.com", None), ("pypi.org", Some("atlas"))]));
assert_eq!(text(&d.allowed()), "# source: swarm\nforge.example\n");
let named = Read::Current(list(&[("github.com", None)]));
sync(&d, named).expect("renders");
assert_eq!(
text(&d.allowed()),
"# source: swarm\nforge.example\ngithub.com\n"
);
assert_eq!(text(&d.agent_allow()), SWARM_MARKER);
}
#[test]
fn a_per_agent_entry_renders_on_top_of_the_seed() {
let d = dir("forge.example\n");
let read = Read::Current(list(&[("pypi.org", Some("atlas"))]));
sync(&d, read).expect("renders");
assert_eq!(text(&d.allowed()), "# source: swarm\nforge.example\n");
assert_eq!(
text(&d.agent_allow()),
"# source: swarm\n\
acl agent_allow_atlas proxy_auth atlas\n\
acl agent_allow_atlas_dst dstdomain pypi.org\n\
http_access allow agent_allow_atlas agent_allow_atlas_dst\n"
);
}
#[test]
fn a_seed_name_the_swarm_repeats_or_squid_allows_anyway_is_written_once() {
// `cache.nixos.org` is always allowed; `.example.com` covers the
// swarm's `api.example.com`.
let d = dir("Forge.Example\n.example.com\ncache.nixos.org\n");
let read = Read::Current(list(&[
("forge.example", None),
("api.example.com", None),
("github.com", None),
]));
sync(&d, read).expect("renders");
assert_eq!(
text(&d.allowed()),
"# source: swarm\n.example.com\nforge.example\ngithub.com\n"
);
}
#[test]
fn no_seed_file_is_no_seed_names() {
let d = dir("");
std::fs::remove_file(d.seed()).expect("remove");
let read = Read::Current(list(&[("github.com", None)]));
sync_with(
read,
&agents(&["atlas"]),
None,
"",
&Files {
allowed: &d.allowed(),
agent_allow: &d.agent_allow(),
},
)
.expect("renders");
assert_eq!(text(&d.allowed()), "# source: swarm\ngithub.com\n");
assert!(text(&d.agent_allow()).contains("acl agent_allow_atlas proxy_auth atlas"));
}
#[test]

View file

@ -5,8 +5,8 @@
# hive-c0re renders. A credential's agent either has the swarm's
# `unrestricted_internet` flag, and may then reach any destination the deny
# rules leave, or reaches only the listed names: the base list, the allow-list
# the swarm keeps (rendered by hive-c0re beside the users file, with this
# host's `allowedDomains` as the list until the swarm has one) and the names
# the swarm keeps plus this host's `allowedDomains` (rendered by hive-c0re
# beside the users file) and the names
# the swarm lists for that agent alone. An agent sends its credential
# once it has fetched it (../agent-modules/egress-proxy.nix); the exporter,
# journal forwarding and the self-check below show the proxy works.
@ -32,8 +32,8 @@ let
# The names squid allows every authenticated user whatever the swarm lists.
# hive-c0re skips an allow-list entry that one of them already covers.
alwaysAllowed = base ++ [ cfg.selfcheck.canary ];
# What `allowedFile` holds until the swarm has an allow-list: this hive's
# own `allowedDomains`, one per line. The canary is in the configuration.
# What `allowedFile` starts as and always includes: this hive's own
# `allowedDomains`, one per line. The canary is in the configuration.
seedFile = pkgs.writeText "squid-allowed-seed" (
lib.concatMapStrings (d: d + "\n") (
lib.unique (lib.subtractLists [ cfg.selfcheck.canary ] cfg.allowedDomains)
@ -254,12 +254,13 @@ in
"crates.io"
];
description = ''
Hostnames squid lets a client reach, by CONNECT or plain HTTP, until the
swarm has an allow-list of its own: the list the swarm's controller
keeps (`/api/egress/allowed-domains`) replaces this one as soon as it
holds an entry or has had one, and this option is the list again only
while the controller has none, for instance because it predates the
feature. hive-c0re writes whichever list applies to `allowedFile`.
Hostnames squid always lets a client reach, by CONNECT or plain HTTP, on
this hive. The entries the swarm's controller keeps
(`/api/egress/allowed-domains`) add to this list and never replace it:
hive-c0re writes these names and the swarm-wide entries to
`allowedFile`. A hive-c0re that predates this rule replaced this list
with the controller's once the controller held an entry, so every hive
must run the new one before the controller creates its first entry.
Each entry matches that exact name, or that name and its subdomains
with a leading dot. Everything else gets a 403, except
for an unrestricted user. The swarm's own service names are not
@ -289,10 +290,11 @@ in
off the swarm to work at all. The defaults are the hosts Claude Code
requires. `api.anthropic.com` serves the model API and the usage
probe, `claude.ai` the account login, and `platform.claude.com` the
OAuth token exchange and refresh. An agent whose `claude` or ACP provider
points elsewhere (`ANTHROPIC_BASE_URL`, an `opencode` provider) needs
that host listed here: the value lives in the agent's own backend
environment, which this host does not read. Swarm-internal names are
OAuth token exchange and refresh. The swarm controller adds the host of an
agent's stored `ANTHROPIC_BASE_URL` to that agent's swarm entries; an
agent on the hand-placed environment file alone, or an `opencode`
provider, needs its host in the swarm UI or in `allowedDomains`.
Swarm-internal names are
not listed, since agents reach them directly. The destination denies
still apply to every entry, and the list is separate from
`allowedDomains`. A name the swarm's allow-list holds that this list
@ -339,9 +341,9 @@ in
description = ''
The names squid lets every restricted user reach, one per line, read as
part of its allow ACL beside the self-check canary. It starts as a copy
of `allowedDomains`. hive-c0re rewrites it every pass: with the swarm's
swarm-wide entries once the controller has a list, otherwise with
`allowedDomains`. Squid is reloaded when it changes. Owned by
of `allowedDomains`. hive-c0re rewrites it every pass: with
`allowedDomains` and the swarm-wide entries once the controller has a
list, otherwise with `allowedDomains` alone. Squid is reloaded when it changes. Owned by
`hive-core` and readable by the `squid` group.
'';
};
@ -368,8 +370,8 @@ in
readOnly = true;
defaultText = lib.literalMD "`allowedDomains`, one per line, minus the self-check canary";
description = ''
The names `allowedFile` starts as and hive-c0re writes to it again while
the swarm has no allow-list. Read-only.
The names `allowedFile` starts as and hive-c0re writes into it on every
pass, with or without a swarm allow-list. Read-only.
'';
};

View file

@ -520,13 +520,13 @@ pkgs.testers.runNixOSTest {
wait_status("other.t.local", RUTH, "403")
assert connect("other.t.local", UMA) == "200", "control: uma is still listed"
with subtest("the swarm's allow-list replaces the hive's names, opens one to every restricted user, and a per-agent rule opens one to that agent alone"):
with subtest("the swarm's allow-list adds to the hive's names, opens one to every restricted user, and a per-agent rule opens one to that agent alone"):
seeded = proxy.succeed(f"cat {ALLOWED}")
assert connect("swarm.t.local", RUTH) == "403", "control: not allowed yet"
assert connect("agentonly.t.local", ATLAS) == "403", "control: not allowed yet"
assert connect("forge.t.local", RUTH) == "200", "control: the hive's own name"
write_allow(
"# source: swarm\nswarm.t.local\n",
"# source: swarm\n" + seeded + "swarm.t.local\n",
"# source: swarm\n"
"acl agent_allow_atlas proxy_auth atlas\n"
"acl agent_allow_atlas_dst dstdomain agentonly.t.local priv10.t.local\n"
@ -537,13 +537,13 @@ pkgs.testers.runNixOSTest {
assert connect("swarm.t.local", ATLAS) == "200", "swarm-wide, another user"
assert connect("agentonly.t.local", RUTH) == "403", "the per-agent name, another restricted user"
assert connect("agentonly.t.local") == "407", "no credential"
assert connect("forge.t.local", RUTH) == "403", "the hive's own name is replaced"
assert connect("forge.t.local", RUTH) == "200", "the hive's own name stays"
assert connect("priv10.t.local", ATLAS) == "403", "a per-agent name that resolves privately"
assert connect("agentonly.t.local", UMA) == "200", "control: unrestricted reaches it anyway"
write_allow(seeded, "")
wait_status("swarm.t.local", RUTH, "403")
wait_status("agentonly.t.local", ATLAS, "403")
assert connect("forge.t.local", RUTH) == "200", "the hive's own name is back"
assert connect("forge.t.local", RUTH) == "200", "the hive's own name after the swarm list is gone"
with subtest("squid starts on a missing list, treats every user as restricted, and recovers when it appears"):
proxy.succeed(f"rm {LIST}")

View file

@ -18,6 +18,11 @@ pub struct AllowedDomain {
#[serde(default, skip_serializing_if = "Option::is_none")]
#[cfg_attr(feature = "schema", schema(example = "atlas"))]
pub agent: Option<String>,
/// Set on an entry the controller keeps from the agent's backend
/// environment (the host of its `ANTHROPIC_BASE_URL`). The list shows it
/// as automatic; adding or removing one is refused.
#[serde(default, skip_serializing_if = "std::ops::Not::not")]
pub derived: bool,
}
/// The swarm's allow-list: every entry, swarm-wide and per agent, sorted.
@ -36,6 +41,7 @@ mod tests {
let swarm_wide = AllowedDomain {
domain: "github.com".to_owned(),
agent: None,
derived: false,
};
let wire = r#"{"domain":"github.com"}"#;
assert_eq!(serde_json::to_string(&swarm_wide).unwrap(), wire);

View file

@ -66,8 +66,9 @@ The swarm's control plane. The swarm UI and `swarmctl` are its clients.
hive's egress proxy, each for every agent or for one, as one value in the
`egress-allow` KV bucket, which this daemon alone writes
(`GET`/`POST`/`DELETE /api/egress/allowed-domains`); every hive reads it and
renders the proxy's allow-list files, and it replaces the list the hive's own
configuration names →
renders the proxy's allow-list files, which add to the names the hive's own
configuration lists. It also keeps a `derived` entry per agent for the host
of its stored backend environment's `ANTHROPIC_BASE_URL` →
[`docs/networking/network.md`](../docs/networking/network.md#egress-proxy-squid).
- **Agent instructions** — each agent's `AGENTS.md` as its harness last
published it into the `agents-md` KV bucket

View file

@ -0,0 +1,820 @@
//! The hosts an agent's backend environment names, kept as derived entries of
//! the egress allow list (`crate::egress_allow`), so a restricted agent whose
//! `ANTHROPIC_BASE_URL` points at a provider other than Anthropic reaches that
//! provider through its hive's proxy.
//!
//! An entry is marked `derived` and a hive renders it like any other. Only the
//! operator's save or delete of the environment through the controller creates
//! one ([`follow_env`]): the agent can create its own `backend-env` leaf while
//! it is empty, so what the store holds is not evidence that an operator chose
//! it. [`spawn_backfill`] therefore only drops: it keeps a derived entry while
//! the stored environment still names its exact host, and never adds one. Each
//! host goes through `egress_allow::checked`, the check an operator's own
//! entry gets.
//!
//! An agent still on the hand-placed `backendEnvironmentFile` alone has no
//! derived entry, and neither does an environment saved before this existed
//! until the operator saves it once more. The opencode preset's provider URL
//! (`acp.opencode.provider.baseUrl`) is host configuration, not part of the
//! environment, so it is never derived either.
use std::collections::{BTreeMap, BTreeSet};
use std::time::Duration;
use problem_details::ProblemDetails;
use swarm_secret_client::backend_env::{self, BackendEnv};
use swarm_wire::egress_allow::{AllowList, Entry, MAX_ENTRIES};
use crate::egress_allow::{base_domains, checked, modify};
use crate::linked_accounts::AccountStore;
use crate::routes::{AppState, error_problem, wanted_error_status};
use crate::schedules::queue;
/// `ANTHROPIC_BASE_URL` and every variable named `<PROVIDER>_BASE_URL`, which
/// is where a provider's address sits for an ACP agent that loads the same
/// file.
fn is_url_variable(name: &str) -> bool {
name == "ANTHROPIC_BASE_URL" || name.ends_with("_BASE_URL")
}
/// The URL variables `env` assigns, as `(name, value)`. A later assignment of a
/// name replaces an earlier one, as systemd's `EnvironmentFile=` does.
fn url_assignments(env: &str) -> BTreeMap<&str, &str> {
let mut found = BTreeMap::new();
for line in env.lines().map(str::trim) {
if line.is_empty() || line.starts_with(['#', ';']) {
continue;
}
let Some((name, value)) = line.split_once('=') else {
continue;
};
let name = name.trim();
if is_url_variable(name) {
found.insert(name, unquoted(value.trim()));
}
}
found
}
fn unquoted(value: &str) -> &str {
for quote in ['"', '\''] {
if let Some(inner) = value
.strip_prefix(quote)
.and_then(|v| v.strip_suffix(quote))
{
return inner;
}
}
value
}
/// The host of `url`, as written: its scheme (`http` or `https`, or none),
/// user information, port, path, query and fragment are dropped. Nothing is
/// validated; the result goes through `egress_allow::checked`.
fn host_of(url: &str) -> Option<&str> {
let rest = match url.split_once("://") {
Some(("http" | "https", rest)) => rest,
Some(_) => return None,
None => url,
};
let authority = rest.split(['/', '?', '#']).next()?;
let authority = authority.rsplit('@').next()?;
Some(match authority.rsplit_once(':') {
Some((host, port)) if port.bytes().all(|b| b.is_ascii_digit()) => host,
_ => authority,
})
}
/// The derived entries for `agent`'s `env`: one per distinct host that passes
/// `egress_allow::checked`. A host that does not (an IP address, `localhost`,
/// a name the base list already allows) is logged by variable name and left
/// out. The value is never logged: a URL can carry credentials.
fn derive(agent: &str, env: &str, base: &[String]) -> Vec<Entry> {
let mut entries: Vec<Entry> = derive_named(agent, env, base)
.into_iter()
.map(|(_, entry)| entry)
.collect();
entries.sort();
entries.dedup();
entries
}
/// [`fn@derive`] with the variable each entry came from, one per variable.
fn derive_named<'a>(agent: &str, env: &'a str, base: &[String]) -> Vec<(&'a str, Entry)> {
let mut entries = Vec::new();
for (name, value) in url_assignments(env) {
let Some(host) = host_of(value) else {
tracing::info!(%agent, variable = name, "backend egress: not an http(s) URL; no entry");
continue;
};
let request = swarm_controller_api::egress::AllowedDomain {
domain: host.to_owned(),
agent: Some(agent.to_owned()),
derived: false,
};
match checked(&request, base) {
Ok(entry) => entries.push((
name,
Entry {
derived: true,
..entry
},
)),
Err(why) => {
tracing::info!(%agent, variable = name, %why, "backend egress: no entry for this host");
}
}
}
entries
}
/// `list` with, for each agent in `desired`, its derived entries replaced by
/// the given ones. Entries of other agents and operator-added entries are
/// untouched. A full list takes no more derived entries.
fn replaced(mut list: AllowList, desired: &BTreeMap<String, Vec<Entry>>) -> AllowList {
list.entries
.retain(|e| !(e.derived && e.agent.as_ref().is_some_and(|a| desired.contains_key(a))));
for entry in desired.values().flatten() {
if list.entries.len() >= MAX_ENTRIES {
tracing::warn!(
domain = %entry.domain,
"backend egress: the allow list is full; no entry for this host"
);
break;
}
list.entries.push(entry.clone());
}
list
}
/// Replace `agent`'s derived entries with those of `env`, or with none when
/// `env` is `None` (the environment was deleted). A host without a queue
/// (`state.status` unset) has no allow list, so nothing happens.
///
/// # Errors
/// A problem naming that the environment is stored but the list was not
/// updated; saving the environment again retries.
pub(crate) async fn follow_env(
state: &AppState,
agent: &str,
env: Option<&str>,
) -> Result<(), ProblemDetails> {
if state.status.is_none() {
return Ok(());
}
let base = base_domains();
let desired = BTreeMap::from([(
agent.to_owned(),
env.map(|text| derive(agent, text, &base))
.unwrap_or_default(),
)]);
let updated = async {
let client = queue(state).map_err(|(_, why)| anyhow::anyhow!(why))?;
modify(&client, |list| replaced(list, &desired)).await
}
.await;
match updated {
Ok(true) => tracing::info!(%agent, "backend egress: derived entries updated"),
Ok(false) => {}
Err(e) => {
tracing::warn!(%agent, error = %format!("{e:#}"), "backend egress: updating the allow list failed");
return Err(error_problem(
wanted_error_status(&e),
&format!(
"the backend environment is stored, but the allowed domains were not \
updated ({e:#}); saving it again retries"
),
));
}
}
Ok(())
}
/// `list` without the derived entries of the agents in `still_named` whose host
/// is not in that agent's set. Nothing is added; entries of agents not in the
/// map and operator-added entries are untouched.
fn pruned(mut list: AllowList, still_named: &BTreeMap<String, BTreeSet<String>>) -> AllowList {
list.entries.retain(|e| {
let Some(hosts) = e.agent.as_ref().and_then(|a| still_named.get(a)) else {
return true;
};
!e.derived || hosts.contains(&e.domain)
});
list
}
/// Drop the derived entries whose host the stored environment no longer names.
///
/// Only the agents that have a derived entry are looked at. An agent whose
/// stored environment is gone loses all of them; one whose environment cannot
/// be read keeps them. A host the stored environment names that is not
/// already derived is left out and logged by variable name, since whoever
/// wrote the store may be the agent itself.
///
/// Answers how many agents could not be read.
///
/// # Errors
/// A queue failure reading or writing the list.
pub(crate) async fn reconcile(
client: &async_nats::Client,
store: &impl AccountStore,
) -> anyhow::Result<usize> {
let Some(current) = crate::egress_allow::current(client).await? else {
return Ok(0);
};
let mut derived: BTreeMap<String, BTreeSet<String>> = BTreeMap::new();
for entry in current.entries.iter().filter(|e| e.derived) {
if let Some(agent) = &entry.agent {
derived
.entry(agent.clone())
.or_default()
.insert(entry.domain.clone());
}
}
let base = base_domains();
let mut still_named = BTreeMap::new();
let mut unreadable = 0;
for (agent, kept) in derived {
let Ok(path) = backend_env::env_path(&agent) else {
continue;
};
match store.read_optional::<BackendEnv>(&path).await {
Ok(stored) => {
let named = stored.map_or_else(Vec::new, |s| {
derive_named(&agent, &s.env, &base)
.into_iter()
.map(|(variable, entry)| (variable.to_owned(), entry.domain))
.collect()
});
for (variable, domain) in &named {
if !kept.contains(domain) {
tracing::warn!(
%agent,
%variable,
"backend egress: the agent's stored env names a host not derived \
(only an operator save derives)"
);
}
}
still_named.insert(agent, named.into_iter().map(|(_, d)| d).collect());
}
Err(e) => {
tracing::warn!(%agent, error = %e, "backend egress: reading the backend env failed; its entries stay");
unreadable += 1;
}
}
}
if modify(client, |list| pruned(list, &still_named)).await? {
tracing::info!(
agents = still_named.len(),
"backend egress: dropped derived entries the stored env no longer names"
);
}
Ok(unreadable)
}
/// Run [`reconcile`] once the queue and the store answer, retrying with a
/// growing wait until a pass reads every agent that has an entry.
pub(crate) fn spawn_backfill(client: async_nats::Client) {
tokio::spawn(async move {
let mut wait = Duration::from_secs(5);
loop {
let pass = async {
swarm_queue_client::ensure_connected(&client)?;
let store = crate::store::connect().await?;
reconcile(&client, &store).await
}
.await;
match pass {
Ok(0) => return,
Ok(n) => tracing::warn!(
unreadable = n,
"backend egress: backfill incomplete; retrying"
),
Err(e) => {
tracing::warn!(error = %format!("{e:#}"), "backend egress: backfill failed; retrying");
}
}
tokio::time::sleep(wait).await;
wait = (wait * 2).min(Duration::from_mins(5));
}
});
}
#[cfg(test)]
mod tests {
use std::sync::Arc;
use super::*;
use crate::agent_limits::tests::declared;
use crate::linked_accounts::tests::FakeStore;
use crate::test_nats::TestNats;
use serde_json::json;
fn base() -> Vec<String> {
vec!["api.anthropic.com".to_owned(), ".claude.ai".to_owned()]
}
fn hosts(env: &str) -> Vec<String> {
derive("atlas", env, &base())
.into_iter()
.map(|e| e.domain)
.collect()
}
#[test]
fn a_url_is_reduced_to_its_host() {
for (value, host) in [
("https://openrouter.ai/api/v1", "openrouter.ai"),
("http://llm.example.com", "llm.example.com"),
("https://LLM.Example.com:8443/v1?x=1#f", "llm.example.com"),
("llm.example.com/v1", "llm.example.com"),
("llm.example.com:8080", "llm.example.com"),
("\"https://llm.example.com/v1\"", "llm.example.com"),
("'https://llm.example.com/v1'", "llm.example.com"),
("https://user:pw@llm.example.com/v1", "llm.example.com"),
] {
assert_eq!(
hosts(&format!("ANTHROPIC_BASE_URL={value}\n")),
[host],
"{value}"
);
}
}
#[test]
fn a_host_the_swarm_would_refuse_gets_no_entry() {
for value in [
"http://10.0.0.5:8080/v1",
"https://127.0.0.1",
"http://[::1]:11434",
"http://localhost:11434/v1",
"https://metadata",
"ftp://llm.example.com",
"https://*.example.com",
"",
"https://",
// The base list already allows it.
"https://api.anthropic.com/v1",
"https://a.claude.ai",
] {
assert_eq!(
hosts(&format!("ANTHROPIC_BASE_URL={value}\n")),
Vec::<String>::new(),
"{value:?}"
);
}
// Control: the same shape with a public name gets one.
assert_eq!(
hosts("ANTHROPIC_BASE_URL=https://llm.example.com\n"),
["llm.example.com"]
);
}
#[test]
fn only_url_variables_count_and_the_last_assignment_wins() {
let env = "# ANTHROPIC_BASE_URL=https://commented.example.com\n\
ANTHROPIC_API_KEY=https://not-a-url-var.example.com\n\
OPENAI_BASE_URL=https://acp.example.org/v1\n\
ANTHROPIC_BASE_URL=https://first.example.com\n\
ANTHROPIC_BASE_URL=https://second.example.com\n";
assert_eq!(hosts(env), ["acp.example.org", "second.example.com"]);
assert_eq!(hosts("ANTHROPIC_API_KEY=sk-x\n"), Vec::<String>::new());
}
#[test]
fn a_derived_entry_is_marked_and_scoped_to_the_agent() {
let entries = derive(
"atlas",
"ANTHROPIC_BASE_URL=https://llm.example.com\n",
&base(),
);
assert_eq!(
entries,
[Entry {
domain: "llm.example.com".to_owned(),
agent: Some("atlas".to_owned()),
derived: true,
}]
);
}
fn manual(domain: &str, agent: &str) -> Entry {
Entry {
domain: domain.to_owned(),
agent: Some(agent.to_owned()),
derived: false,
}
}
fn derived(domain: &str, agent: &str) -> Entry {
Entry {
derived: true,
..manual(domain, agent)
}
}
#[test]
fn replacing_replaces_only_the_named_agents_derived_entries() {
let list = AllowList {
entries: vec![
derived("old.example.com", "atlas"),
manual("old.example.com", "atlas"),
manual("github.com", "atlas"),
derived("other.example.com", "iris"),
],
};
let desired = BTreeMap::from([(
"atlas".to_owned(),
vec![derived("new.example.com", "atlas")],
)]);
let after = replaced(list, &desired);
assert_eq!(
after.entries,
[
manual("old.example.com", "atlas"),
manual("github.com", "atlas"),
derived("other.example.com", "iris"),
derived("new.example.com", "atlas"),
]
);
}
/// The state of a test swarm whose `pr1ma` declares `atlas`, with `store`
/// as the secret store.
async fn swarm(client: &async_nats::Client, store: &Arc<FakeStore>) -> AppState {
AppState {
accounts: crate::linked_accounts::Accounts::Fake(Arc::clone(store)),
..declared(client).await
}
}
async fn list_of(client: &async_nats::Client) -> Vec<Entry> {
crate::egress_allow::current(client)
.await
.expect("the queue answers")
.map(|l| {
let mut e = l.entries;
e.sort();
e
})
.unwrap_or_default()
}
async fn put(state: &AppState, agent: &str, env: &str) {
crate::backend_env::put_backend_env(
axum::extract::State(state.clone()),
axum::extract::Path(("pr1ma".to_owned(), agent.to_owned())),
axum::Json(serde_json::from_value(json!({"env": env})).expect("a request body")),
)
.await
.expect("stored");
}
#[tokio::test]
async fn saving_changing_and_deleting_the_env_moves_the_derived_entries() {
let nats = TestNats::start();
let client = nats.client().await;
let store = Arc::default();
let state = swarm(&client, &store).await;
put(
&state,
"atlas",
"ANTHROPIC_BASE_URL=https://one.example.com/v1\n",
)
.await;
assert_eq!(
list_of(&client).await,
[derived("one.example.com", "atlas")]
);
// A change replaces the host, an operator's own entry stays.
let _ = crate::egress_allow::add_allowed_domain(
axum::extract::State(state.clone()),
axum::Json(swarm_controller_api::egress::AllowedDomain {
domain: "github.com".to_owned(),
agent: Some("atlas".to_owned()),
derived: false,
}),
)
.await
.expect("manual entry");
put(
&state,
"atlas",
"ANTHROPIC_BASE_URL=https://two.example.com/v1\n",
)
.await;
assert_eq!(
list_of(&client).await,
[
manual("github.com", "atlas"),
derived("two.example.com", "atlas")
]
);
// Removing the variable removes the entry.
put(&state, "atlas", "ANTHROPIC_API_KEY=sk-live\n").await;
assert_eq!(list_of(&client).await, [manual("github.com", "atlas")]);
// Deleting the env takes the entries of the env it held.
put(
&state,
"atlas",
"ANTHROPIC_BASE_URL=https://three.example.com\n",
)
.await;
crate::linked_accounts::delete_backend_env(
axum::extract::State(state.clone()),
axum::extract::Path(("pr1ma".to_owned(), "atlas".to_owned())),
)
.await
.expect("deleted");
assert_eq!(list_of(&client).await, [manual("github.com", "atlas")]);
}
#[tokio::test]
async fn an_env_with_no_host_never_creates_the_list() {
let nats = TestNats::start();
let client = nats.client().await;
let store = Arc::default();
let state = swarm(&client, &store).await;
put(&state, "atlas", "ANTHROPIC_API_KEY=sk-live\n").await;
assert!(
swarm_queue_client::egress_allow::open(&client)
.await
.expect("the queue answers")
.is_none(),
"no bucket: nothing was written"
);
// Control: a host does create it.
put(
&state,
"atlas",
"ANTHROPIC_BASE_URL=https://llm.example.com\n",
)
.await;
assert_eq!(list_of(&client).await.len(), 1);
}
const OPERATOR: &str = "ANTHROPIC_BASE_URL=https://llm.example.com/v1\n";
fn leaf(agent: &str) -> String {
format!("swarm/agents/{agent}/backend-env")
}
async fn write_leaf(store: &FakeStore, agent: &str, env: &str) {
store
.write(
&leaf(agent),
&BackendEnv {
env: env.to_owned(),
},
)
.await
.expect("written");
}
/// The agent creates its own leaf while it is empty, so a stored value is
/// not evidence of an operator's choice: the backfill adds nothing for it.
#[tokio::test]
async fn a_leaf_created_outside_the_route_yields_no_entry() {
let nats = TestNats::start();
let client = nats.client().await;
let store = Arc::new(FakeStore::default());
let state = swarm(&client, &store).await;
write_leaf(
&store,
"atlas",
"ANTHROPIC_BASE_URL=https://evil.example.net\n",
)
.await;
// With no list at all, the pass creates nothing.
assert_eq!(reconcile(&client, &*store).await.expect("pass"), 0);
assert!(
swarm_queue_client::egress_allow::open(&client)
.await
.expect("the queue answers")
.is_none()
);
// With a list that holds only an operator's entry, nothing is added.
modify(&client, |mut list| {
list.entries.push(manual("github.com", "atlas"));
list
})
.await
.expect("seed");
assert_eq!(reconcile(&client, &*store).await.expect("pass"), 0);
assert_eq!(list_of(&client).await, [manual("github.com", "atlas")]);
// Control: the operator's save of the same text derives the host.
put(
&state,
"atlas",
"ANTHROPIC_BASE_URL=https://evil.example.net\n",
)
.await;
assert_eq!(
list_of(&client).await,
[
derived("evil.example.net", "atlas"),
manual("github.com", "atlas")
]
);
}
/// A pass keeps an entry whose exact host the stored env still names,
/// drops one it no longer names, and never adds the new host.
#[tokio::test]
async fn the_backfill_keeps_a_named_host_drops_a_stale_one_and_adds_nothing() {
let nats = TestNats::start();
let client = nats.client().await;
let store = Arc::new(FakeStore::default());
let state = swarm(&client, &store).await;
put(&state, "atlas", OPERATOR).await;
let entry = derived("llm.example.com", "atlas");
// Control: unchanged content keeps the entry, and a second pass writes
// nothing.
assert_eq!(reconcile(&client, &*store).await.expect("pass"), 0);
assert_eq!(list_of(&client).await, [entry]);
let before = published(&client).await;
reconcile(&client, &*store).await.expect("again");
assert_eq!(published(&client).await, before);
// The stored text changed outside the route: the old host goes, the
// new one does not come.
write_leaf(
&store,
"atlas",
"ANTHROPIC_BASE_URL=https://other.example.net\n",
)
.await;
reconcile(&client, &*store).await.expect("pass");
assert_eq!(list_of(&client).await, []);
}
/// The leaf is deleted outside the controller and created again by the
/// agent with another host before the pass: the old entry goes and the new
/// host is not derived. A leaf that is simply gone drops its entries too.
#[tokio::test]
async fn a_leaf_deleted_and_recreated_by_the_agent_loses_the_entry_and_gains_none() {
let nats = TestNats::start();
let client = nats.client().await;
let state = swarm(&client, &Arc::default()).await;
put(&state, "atlas", OPERATOR).await;
put(&state, "iris", OPERATOR).await;
assert_eq!(list_of(&client).await.len(), 2, "control: both derived");
let recreated = FakeStore::default();
write_leaf(
&recreated,
"atlas",
"ANTHROPIC_BASE_URL=https://evil.example.net\n",
)
.await;
// iris's leaf is gone, atlas's was created again by atlas.
reconcile(&client, &recreated).await.expect("pass");
assert_eq!(list_of(&client).await, []);
}
#[tokio::test]
async fn an_unreadable_env_keeps_its_entries() {
let nats = TestNats::start();
let client = nats.client().await;
let state = swarm(&client, &Arc::default()).await;
put(
&state,
"atlas",
"ANTHROPIC_BASE_URL=https://llm.example.com\n",
)
.await;
let unreadable = reconcile(&client, &FakeStore::denying())
.await
.expect("pass");
assert_eq!(unreadable, 1);
assert_eq!(
list_of(&client).await,
[derived("llm.example.com", "atlas")]
);
}
async fn published(client: &async_nats::Client) -> Option<String> {
swarm_queue_client::egress_allow::open_or_create(client)
.await
.expect("bucket")
.get(swarm_wire::egress_allow::KEY)
.await
.expect("read")
.map(|v| String::from_utf8(v.to_vec()).expect("utf-8"))
}
#[tokio::test]
async fn an_operator_cannot_add_or_remove_a_derived_entry() {
let nats = TestNats::start();
let client = nats.client().await;
let store = Arc::default();
let state = swarm(&client, &store).await;
put(
&state,
"atlas",
"ANTHROPIC_BASE_URL=https://llm.example.com\n",
)
.await;
let request = |derived: bool| swarm_controller_api::egress::AllowedDomain {
domain: "llm.example.com".to_owned(),
agent: Some("atlas".to_owned()),
derived,
};
for flagged in [true, false] {
let problem = crate::egress_allow::remove_allowed_domain(
axum::extract::State(state.clone()),
axum::Json(request(flagged)),
)
.await
.expect_err("refused");
assert_eq!(
problem.status,
Some(axum::http::StatusCode::CONFLICT),
"{problem:?}"
);
}
let problem = crate::egress_allow::add_allowed_domain(
axum::extract::State(state.clone()),
axum::Json(request(true)),
)
.await
.expect_err("refused");
assert_eq!(problem.status, Some(axum::http::StatusCode::BAD_REQUEST));
assert_eq!(
list_of(&client).await,
[derived("llm.example.com", "atlas")]
);
// Control: the same name added by the operator is a separate entry that
// the operator can remove, and the derived one stays.
let manual_request = request(false);
let _ = crate::egress_allow::add_allowed_domain(
axum::extract::State(state.clone()),
axum::Json(manual_request.clone()),
)
.await
.expect("a manual entry");
let _ = crate::egress_allow::remove_allowed_domain(
axum::extract::State(state.clone()),
axum::Json(manual_request),
)
.await
.expect("removes the manual one");
assert_eq!(
list_of(&client).await,
[derived("llm.example.com", "atlas")]
);
}
#[tokio::test]
async fn destroying_an_agent_drops_all_its_entries_and_only_its_own() {
let nats = TestNats::start();
let client = nats.client().await;
let state = swarm(&client, &Arc::default()).await;
put(
&state,
"atlas",
"ANTHROPIC_BASE_URL=https://llm.example.com\n",
)
.await;
modify(&client, |mut list| {
list.entries.push(manual("github.com", "atlas"));
list.entries.push(manual("crates.io", "iris"));
list
})
.await
.expect("seed");
crate::egress_allow::clear_on_destroy(&state, "pr1ma", "atlas").await;
assert_eq!(list_of(&client).await, [manual("crates.io", "iris")]);
}
#[tokio::test]
async fn a_host_without_a_queue_stores_the_env_and_skips_the_list() {
let nats = TestNats::start();
let client = nats.client().await;
let store = Arc::new(FakeStore::default());
let mut state = swarm(&client, &store).await;
state.status = None;
put(
&state,
"atlas",
"ANTHROPIC_BASE_URL=https://llm.example.com\n",
)
.await;
assert_eq!(store.written(), ["swarm/agents/atlas/backend-env"]);
assert_eq!(list_of(&client).await, []);
}
}

View file

@ -76,6 +76,8 @@ pub async fn put_backend_env(
error_problem(StatusCode::INTERNAL_SERVER_ERROR, &e.to_string())
})?;
crate::backend_egress::follow_env(&state, &agent, Some(&stored.env)).await?;
tracing::info!(%hive, %agent, "backend env stored");
Ok(StatusCode::NO_CONTENT)
}

View file

@ -5,8 +5,9 @@
//! (`swarm_wire::egress_allow`), which this daemon alone writes. Every hive
//! reads it and renders the entries for the agents it runs: a swarm-wide entry
//! joins the proxy's allow list for everyone, an entry with an agent is that
//! agent's alone. Once the key exists it replaces the list a hive's own nix
//! configuration names, so an empty list is a list.
//! agent's alone. A hive always allows the names its own nix configuration
//! lists (`services.hyperhive.squid.allowedDomains`) as well, so these entries
//! only add to them.
//!
//! Each name goes into the proxy's configuration, so it is checked here with
//! `swarm_wire::egress_allow::normalize_domain` (the hive checks again) and
@ -28,6 +29,10 @@ use crate::schedules::{live, queue, read_or_default, refused, write_revision};
/// `wanted::MAX_ATTEMPTS`.
const MAX_ATTEMPTS: usize = 5;
/// Why an operator can neither add nor remove an entry the controller derived.
const DERIVED_REFUSED: &str = "this entry is kept automatically from the agent's backend environment; \
change or remove that variable instead";
/// Set from the hive module's `baseAllowedDomains` default by
/// `nix/host-modules/swarm-controller.nix`: the names, separated by spaces,
/// that every hive's proxy lets every authenticated agent reach. Unset means
@ -35,7 +40,7 @@ const MAX_ATTEMPTS: usize = 5;
/// the entries its own base list covers.
pub(crate) const ENV_BASE_DOMAINS: &str = "SWARM_CONTROLLER_SQUID_BASE_DOMAINS";
fn base_domains() -> Vec<String> {
pub(crate) fn base_domains() -> Vec<String> {
std::env::var(ENV_BASE_DOMAINS)
.unwrap_or_default()
.split_whitespace()
@ -60,7 +65,11 @@ pub(crate) fn checked(req: &AllowedDomain, base: &[String]) -> Result<Entry, Str
.map_err(|reason| format!("agent {a:?}: {reason}"))
})
.transpose()?;
Ok(Entry { domain, agent })
Ok(Entry {
domain,
agent,
derived: false,
})
}
/// What adding an entry to a list came to.
@ -95,6 +104,7 @@ fn view(list: AllowList) -> AllowedDomains {
.map(|e| AllowedDomain {
domain: e.domain,
agent: e.agent,
derived: e.derived,
})
.collect(),
}
@ -118,7 +128,7 @@ async fn open_or_create(
path = "/api/egress/allowed-domains",
responses(
(status = 200, description = "every entry, swarm-wide and per agent, sorted; empty \
when the swarm has not set a list, in which case each hive keeps its own",
when the swarm has not set a list",
body = AllowedDomains),
(status = 503, description = "no swarm queue is configured on this host, or it is \
not connected (problem+json)", body = String),
@ -145,10 +155,8 @@ pub(crate) async fn get_allowed_domains(
/// Allow one destination, for every agent or for one.
///
/// Adding an entry that is already there changes nothing. The first entry
/// makes the swarm's list replace each hive's own, so a name the hive's nix
/// configuration allows and this list lacks stops being allowed once hives read
/// it.
/// Adding an entry that is already there changes nothing. The entry adds to
/// the names each hive's nix configuration already allows.
#[utoipa::path(
post,
path = "/api/egress/allowed-domains",
@ -158,7 +166,7 @@ pub(crate) async fn get_allowed_domains(
(status = 400, description = "the domain is not a DNS name (an IP address, a \
wildcard, a single label and a port are refused), the proxy already allows \
it to everyone, the agent is not an identifier or not declared on any hive, \
or the list is full (problem+json)", body = String),
or the list is full, or the entry is marked derived (problem+json)", body = String),
(status = 422, description = "an unknown field"),
(status = 503, description = "no swarm queue is configured on this host, or it is \
not connected (problem+json)", body = String),
@ -171,6 +179,9 @@ pub(crate) async fn add_allowed_domain(
State(state): State<AppState>,
Json(req): Json<AllowedDomain>,
) -> Result<Json<AllowedDomains>, ProblemDetails> {
if req.derived {
return Err(error_problem(StatusCode::BAD_REQUEST, DERIVED_REFUSED));
}
let entry = checked(&req, &base_domains())
.map_err(|why| error_problem(StatusCode::BAD_REQUEST, &why))?;
let client = queue(&state).map_err(refused)?;
@ -209,8 +220,8 @@ pub(crate) async fn add_allowed_domain(
/// Stop allowing one destination for the scope it was added with.
///
/// Removing an entry that is not there changes nothing. Removing the last one
/// leaves an empty list, which still replaces each hive's own.
/// Removing an entry that is not there changes nothing. A name a hive's nix
/// configuration lists stays allowed on that hive.
#[utoipa::path(
delete,
path = "/api/egress/allowed-domains",
@ -219,6 +230,9 @@ pub(crate) async fn add_allowed_domain(
(status = 200, description = "the whole list after the removal", body = AllowedDomains),
(status = 400, description = "the domain is not a DNS name, or the agent is not an \
identifier (problem+json)", body = String),
(status = 409, description = "the entry is one the controller keeps from the agent's \
backend environment, which an operator cannot remove (problem+json)",
body = String),
(status = 422, description = "an unknown field"),
(status = 503, description = "no swarm queue is configured on this host, or it is \
not connected (problem+json)", body = String),
@ -231,6 +245,9 @@ pub(crate) async fn remove_allowed_domain(
State(state): State<AppState>,
Json(req): Json<AllowedDomain>,
) -> Result<Json<AllowedDomains>, ProblemDetails> {
if req.derived {
return Err(error_problem(StatusCode::CONFLICT, DERIVED_REFUSED));
}
// The base list is not consulted: an entry stored before a name joined it
// can still be removed.
let entry = checked(&req, &[]).map_err(|why| error_problem(StatusCode::BAD_REQUEST, &why))?;
@ -241,6 +258,13 @@ pub(crate) async fn remove_allowed_domain(
.await
.map_err(|e| queue_problem(&e))?;
if !list.entries.contains(&entry) {
if list
.entries
.iter()
.any(|e| e.derived && e.domain == entry.domain && e.agent == entry.agent)
{
return Err(error_problem(StatusCode::CONFLICT, DERIVED_REFUSED));
}
return Ok(Json(view(list)));
}
list.entries.retain(|e| *e != entry);
@ -259,6 +283,86 @@ pub(crate) async fn remove_allowed_domain(
)))
}
/// The stored list, or `None` when no one has written it.
///
/// # Errors
/// A queue failure.
pub(crate) async fn current(client: &async_nats::Client) -> anyhow::Result<Option<AllowList>> {
let Some(store) = egress_allow::open(client).await? else {
return Ok(None);
};
let (list, _, _) = read_or_default::<AllowList>(&store, KEY).await?;
Ok(Some(list))
}
/// Re-read the list and write `change` of it, retrying a lost write, until it
/// is written or `change` leaves it as it is. Answers whether a write landed.
///
/// A list that was never written is written only when `change` makes it
/// non-empty, so a change that adds nothing creates no bucket or key.
///
/// # Errors
/// A queue failure, or `MAX_ATTEMPTS` conflicting writes.
pub(crate) async fn modify(
client: &async_nats::Client,
change: impl Fn(AllowList) -> AllowList,
) -> anyhow::Result<bool> {
let mut store = egress_allow::open(client).await?;
for _ in 0..MAX_ATTEMPTS {
let (mut list, revision) = match &store {
Some(store) => {
let (list, revision, _) = read_or_default::<AllowList>(store, KEY).await?;
(list, revision)
}
None => (AllowList::default(), None),
};
list.entries.sort();
let mut changed = change(list.clone());
changed.entries.sort();
changed.entries.dedup();
if changed == list && (revision.is_some() || changed.entries.is_empty()) {
return Ok(false);
}
if store.is_none() {
store = Some(egress_allow::open_or_create(client).await?);
}
let target = store.as_ref().expect("opened above");
if write_revision(target, KEY, &changed, revision)
.await?
.is_some()
{
return Ok(true);
}
}
anyhow::bail!("gave up after {MAX_ATTEMPTS} conflicting writes")
}
/// Drop every entry of a destroyed agent, derived and operator-added: a name
/// is free again once its agent is destroyed, and the next agent of that name
/// would inherit what it left. A failure is logged and does not stop the
/// destroy, as in `agent_permissions::clear_on_destroy`.
pub(crate) async fn clear_on_destroy(state: &AppState, hive: &str, agent: &str) {
let cleared = async {
let client = queue(state).map_err(|(_, why)| anyhow::anyhow!(why))?;
modify(&client, |mut list| {
list.entries.retain(|e| e.agent.as_deref() != Some(agent));
list
})
.await
}
.await;
match cleared {
Ok(_) => tracing::info!(hive, agent, "destroyed agent's allowed domains cleared"),
Err(e) => tracing::error!(
hive,
agent,
error = %format!("{e:#}"),
"clearing this agent's allowed domains FAILED; it was destroyed anyway, so a new \
agent of the same name inherits them. Declaring the agent destroyed again re-runs this."
),
}
}
#[cfg(test)]
mod tests {
use super::*;
@ -269,6 +373,7 @@ mod tests {
AllowedDomain {
domain: domain.to_owned(),
agent: agent.map(str::to_owned),
derived: false,
}
}
@ -434,7 +539,7 @@ mod tests {
assert_eq!(
published(&client).await.as_deref(),
Some(r#"{"entries":[]}"#),
"the last removal leaves an empty list, which still replaces a hive's own"
"the last removal leaves an empty list"
);
}
@ -445,12 +550,14 @@ mod tests {
.map(|i| Entry {
domain: format!("host{i}.example.com"),
agent: None,
derived: false,
})
.collect(),
};
let new = Entry {
domain: "github.com".to_owned(),
agent: None,
derived: false,
};
let why = added(full.clone(), &new).err().expect("a full list");
assert!(why.contains("1000"), "{why}");
@ -469,7 +576,7 @@ mod tests {
}
/// A read before anything was added answers an empty list and creates no
/// bucket, so a hive keeps its own list.
/// bucket.
#[tokio::test]
async fn a_read_before_any_entry_answers_empty_and_creates_no_bucket() {
let nats = TestNats::start();

View file

@ -621,6 +621,8 @@ pub async fn delete_backend_env(
e.problem()
})?;
crate::backend_egress::follow_env(&state, &agent, None).await?;
tracing::info!(%hive, %agent, "backend env deleted");
Ok(StatusCode::NO_CONTENT)
}

View file

@ -39,6 +39,7 @@ mod alerts;
mod archetypes;
mod attic_copy;
mod auth;
mod backend_egress;
mod backend_env;
mod builder;
mod cache_miss;

View file

@ -17,7 +17,7 @@ use crate::routes::hives::HiveEntry;
use crate::routes::{AppState, StatusUnavailable, error_problem, wanted_error_status};
use crate::{
agent_archetype, agent_identity, agent_limits, agent_permissions, agent_status,
agents_md_initial, auth, inference, schedules, wanted,
agents_md_initial, auth, egress_allow, inference, schedules, wanted,
};
/// The swarm's agent roster — every agent the swarm holds an identity for.
@ -214,6 +214,7 @@ pub(crate) async fn set_agent_state(
agent_permissions::clear_on_destroy(&state, &hive, &agent).await;
agent_archetype::clear_on_destroy(&state, &hive, &agent).await;
agents_md_initial::clear_on_destroy(&state, &hive, &agent).await;
egress_allow::clear_on_destroy(&state, &hive, &agent).await;
}
Ok(Json(render(&declaration)))
}

View file

@ -375,7 +375,8 @@ fn spawn_subagent_streams(status: Option<&Arc<status::StatusReader>>, hives: &[H
}
/// The per-agent background passes: subagent and terminal streams, and
/// schedule keys, and the permissions bucket and its import, and the
/// schedule keys, and the permissions bucket and its import, the allowed
/// domains derived from backend environments, and the
/// archetypes agents are created from. Beside them, the
/// copy of the cache facts to each hive, and the proxy credentials squid checks.
/// Lifted out of `main` for `clippy::too_many_lines`.
@ -386,6 +387,7 @@ pub(crate) fn spawn_per_agent_upkeep(
spawn_subagent_streams(status, hives);
spawn_schedule_reconcile(status, hives);
spawn_agent_permissions(status, hives);
spawn_backend_egress_backfill(status);
crate::attic_copy::spawn(hives.iter().map(|h| h.name.clone()).collect());
crate::proxy_credential::spawn(hives.iter().map(|h| h.name.clone()).collect());
if let Some(status) = status {
@ -403,6 +405,14 @@ fn spawn_agent_permissions(status: Option<&Arc<status::StatusReader>>, hives: &[
agent_permissions::spawn(status.queue_client(), wanted, hives);
}
/// Drop the allowed-domain entries derived from backend environments that no
/// longer name their host, when a swarm queue is wired up.
fn spawn_backend_egress_backfill(status: Option<&Arc<status::StatusReader>>) {
if let Some(status) = status {
crate::backend_egress::spawn_backfill(status.queue_client());
}
}
/// Start rewriting every agent's schedule key from the definitions when a
/// swarm queue is wired up. Same shape as [`spawn_subagent_streams`].
fn spawn_schedule_reconcile(status: Option<&Arc<status::StatusReader>>, hives: &[HiveEntry]) {

View file

@ -44,6 +44,12 @@ pub struct Entry {
/// The agent this entry is for; absent means every agent in the swarm.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub agent: Option<String>,
/// The controller derived this entry from the agent's backend environment
/// and replaces it whenever that changes; an operator can neither add nor
/// remove one. A hive renders it like any other entry, and a reader that
/// predates the field ignores it.
#[serde(default, skip_serializing_if = "std::ops::Not::not")]
pub derived: bool,
}
/// The value under [`BUCKET`]/[`KEY`]. No `deny_unknown_fields`: a field a
@ -222,10 +228,12 @@ mod tests {
Entry {
domain: "github.com".to_owned(),
agent: None,
derived: false,
},
Entry {
domain: "crates.io".to_owned(),
agent: Some("atlas".to_owned()),
derived: false,
},
],
};
@ -238,4 +246,23 @@ mod tests {
AllowList::default()
);
}
/// A hive that predates `derived` decodes the entry through this same
/// tolerance: it keeps `domain` and `agent` and drops the rest.
#[test]
fn a_derived_entry_round_trips_and_an_entry_tolerates_fields_it_does_not_know() {
let derived = Entry {
domain: "llm.example.com".to_owned(),
agent: Some("atlas".to_owned()),
derived: true,
};
let wire = r#"{"domain":"llm.example.com","agent":"atlas","derived":true}"#;
assert_eq!(serde_json::to_string(&derived).unwrap(), wire);
assert_eq!(serde_json::from_str::<Entry>(wire).unwrap(), derived);
// Absent means a manual entry; an unknown field is dropped.
let manual: Entry =
serde_json::from_str(r#"{"domain":"a.io","agent":"atlas","later":1}"#).unwrap();
assert!(!manual.derived);
assert_eq!(manual.domain, "a.io");
}
}