diff --git a/docs/networking/network.md b/docs/networking/network.md index 74ff49a9..9bfff2bf 100644 --- a/docs/networking/network.md +++ b/docs/networking/network.md @@ -45,7 +45,7 @@ that touches it. | container | netns | IPv4 | listens / reached via | | ------------------------- | ----------------------- | -------------- | ----------------------------------------------------------------------------------------------- | | gateway (host) | host | host addresses | nginx `:80`/`:443` (every vhost); dnsmasq `bridgeIp:53` + DHCP `:67` on the bridge | -| egress proxy (host) | host | `bridgeIp` | squid `bridgeIp:3128`; no agent uses it yet | +| egress proxy (host) | host | `bridgeIp` | squid `bridgeIp:3128`; agents with a credential send HTTP and HTTPS through it | | `hive-forge` | host (shared) | host addresses | forgejo `:3000` http, `:2222` git-ssh; fronted by the `forge.` vhost | | `hive-matrix` | private, veth on bridge | DHCP pool | tuwunel on UDS `/run/hive-matrix/tuwunel.sock` → the matrix vhost; out to store, SSO, peers | | `swarm-ci-runner` | private, veth on bridge | DHCP pool | outbound only (runner → forge); no inbound surface | @@ -212,9 +212,11 @@ DNS/gateway), so only expose services safe for any agent to reach. Every hive host runs squid in the host netns, listening on `:3128` (`nix/host-modules/hive-squid.nix`). The port is open on the bridge interface -only. **No agent uses it yet**: none sends a credential, and agent traffic -still goes straight out through the NAT. The proxy goes first so you can check -its metrics and logs before anything reroutes. +only. An agent sends its outbound HTTP and HTTPS through it once its hive says +the proxy is ready and it has fetched its proxy credential (see _Agents and the +proxy_ below). Any other agent goes straight out through the NAT, as before, +and so does any program that ignores the proxy variables: nothing blocks direct +egress yet. Squid filters on the destination hostname: a `CONNECT` target or a plain-HTTP host. It never decrypts TLS, so it holds no signing key. It caches nothing. @@ -233,6 +235,42 @@ host. It never decrypts TLS, so it holds no signing key. It caches nothing. A request without a valid credential gets a 407, once the destination checks below have passed it. +- **Agents and the proxy.** An agent uses the proxy only when its hive says + this host's squid is ready for it. `hive-c0re` puts two facts in every + container's `/etc/hyperhive-hive-facts`, only on a hive that holds a store + identity: `HIVE_SQUID_PROXY` (`services.hyperhive.squid.listen`, the bridge IP + and port) and `HIVE_EGRESS_NO_PROXY` (the direct hosts below). A hive that + predates them writes neither, and its agents go direct. + `hive-agent-proxy-credential.service` (`nix/agent-modules/egress-proxy.nix`) + runs in each agent's container, as the agent user, at start and every 10 + minutes. With both facts present it logs in to the secret store with the + agent's own certificate, reads `swarm/agents//proxy-credential` and + writes `/run/hive-agent-proxy-credential/proxy.env` (`0600`) with + `HTTPS_PROXY`, `HTTP_PROXY`, their lower-case forms, `NODE_USE_ENV_PROXY=1` + and `NO_PROXY`/`no_proxy`. The proxy URL is + `http://:@:3128`; Claude Code, Node, curl, git + and reqwest send the userinfo as `Proxy-Authorization`. The harness, its bash + tool, the subagents, the MCP servers and the pollers load the file with + `EnvironmentFile=-` when they start. Until the facts and the credential both + exist nothing writes the file, so those units have no proxy variable and go + direct; a failed read keeps the file already in place. Nothing restarts on + deploy: an agent takes the proxy at its next start, which is its next rebuild + or restart. The credential is in the environment of every process those units + start, so `env` in the agent's shell prints it. Login shells + (`nixos-container root-login`, SSH) don't load the file. +- **Direct destinations.** `HIVE_EGRESS_NO_PROXY` is the hosts this agent must + reach without the proxy, and `NO_PROXY` is `localhost,127.0.0.1,::1` followed + by that list. `hive-c0re` computes it from the host's options: the bridge IP, + the hive's domain (`services.hyperhive.domain`), the secret store and queue + names (`services.hyperhive.swarm.bao.domain`, `swarm.nats.domain`), + `services.hyperhive.swarm.serviceDomains` (the swarm UI and controller apex, + the forge, matrix, attic, logs, metrics and otel names) and every name the + gateway answers with the bridge IP (`services.hyperhive.gateway.localNames`). + Squid refuses every destination that resolves to the bridge, and the hive's + resolver answers the bridge address for those names, so they never go through + the proxy. A bare name matches itself and its subdomains. Node's `NO_PROXY` + has no CIDR support, so the list holds the bridge as an address. + - **Two lists.** The credential decides the policy. `hive-c0re` writes a second file beside the users file, `services.hyperhive.squid.unrestrictedFile` (`/var/lib/hive-squid/unrestricted`, `hive-core:squid`, `0640`): the names of diff --git a/docs/process/conventions.md b/docs/process/conventions.md index 61bd3141..c876f286 100644 --- a/docs/process/conventions.md +++ b/docs/process/conventions.md @@ -43,10 +43,10 @@ agent's build, and neither do the hive's other per-host values: the dashboard port (`HIVE_DASHBOARD_PORT`), the pinned claude-code (`HYPERHIVE_CLAUDE_CODE_PATH`), extra OTEL labels (`HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES`) and the queue's address -(`HIVE_AGENT_NATS_URL`, from `deploy.hive-controller.queue.agentNatsUrl`). `hive-priv` writes them to +(`HIVE_AGENT_NATS_URL`, from `deploy.hive-controller.queue.agentNatsUrl`) and the egress proxy's address and direct hosts (`HIVE_SQUID_PROXY`, `HIVE_EGRESS_NO_PROXY`, see [the egress proxy](../networking/network.md#egress-proxy-squid)). `hive-priv` writes them to `/etc/hyperhive-hive-facts` before every container start, leaving out any the hive doesn't set; the agent then uses its own default. The agent's environment -generator (`nix/agent-modules/hive-facts.nix`) passes them to every unit, +generator (`nix/agent-modules/hive-facts.nix`) passes all but the two egress facts, which only the proxy fetch reads, to every unit, including `hivectl choom` sessions, and `/etc/set-environment` exports them in login shells that no unit starts (`nixos-container root-login`). None is part of an agent's closure, and an agent started on another hive reports that hive. @@ -323,7 +323,7 @@ that value. Read it with `swarm_queue_client::agent_permissions::unrestricted`. The hive reads it for the agent's squid policy: [the egress proxy](../networking/network.md#egress-proxy-squid) lets an unrestricted agent's credential reach any destination, and holds the -others to the allow-list. No agent sends a credential yet. +others to the allow-list. An agent sends its credential once it has fetched it, from its next start. **How a change applies** — - The agent's harness reads its own key every 10 seconds and builds every diff --git a/docs/swarm/credentials-reference.md b/docs/swarm/credentials-reference.md index 77657ec6..57d3134e 100644 --- a/docs/swarm/credentials-reference.md +++ b/docs/swarm/credentials-reference.md @@ -36,7 +36,7 @@ the rest of the cell says how. | `swarm/agents//claude-credentials` | `swarm-claude-login`, when you log the agent in from its card; one version per login | the agent, while it waits for a login | ❌ log in again | ✅ every 30 seconds while waiting | | `swarm/hives//matrix/sender-token` | `swarm-controller`, every 5 minutes | `swarm-controller` and the hive | ✅ when missing or rejected by the homeserver | ✅ the hive re-reads on every sweep | | `swarm/hives//forge/token` | `swarm-controller`, every 5 minutes | `swarm-controller` and the hive | ✅ when missing or no longer matching the forge | ✅ read on every forge call | -| `swarm/agents//proxy-credential` | `swarm-controller`: 32 random bytes, at start and every 5 minutes for every agent that holds a store identity | the hive (`hive-c0re`); the agent can read it, nothing sends it yet | ❌ never replaced: written only while the path holds nothing | ✅ `hive-c0re` re-reads it every 30 seconds | +| `swarm/agents//proxy-credential` | `swarm-controller`: 32 random bytes, at start and every 5 minutes for every agent that holds a store identity | the hive (`hive-c0re`) and the agent, which fetches it | ❌ never replaced: written only while the path holds nothing | ✅ `hive-c0re` re-reads it every 30 seconds | | `swarm/hives//squid-selfcheck` | `swarm-controller`, in the same pass as the agents' credentials | the hive: `hive-c0re` and `squid-selfcheck-credential` (root) | ❌ never replaced: written only while the path holds nothing | ✅ `hive-c0re` every 30 seconds, the root unit every 10 minutes | | `swarm/hives//oidc/client` | authelia, first boot; `swarm-secret-publish` copies it in | the hive | ❌ minted once; `swarm-secret-publish` republishes a re-mint | ✅ re-read when authelia refuses it | | `swarm/services//oidc/client` | authelia | that service | must be stated | must be stated | @@ -130,6 +130,7 @@ pieces fit, and the reasoning behind them, is in [`credentials.md`](credentials. | the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | the swarm secret store at `swarm/controller/swarm-controller/forge/token`, written by the mint under the forge publisher's leaf, which the forge container loads as a credential; the controller holds it in memory only. `/var/lib/forgejo/swarm-controller-token-name` in the forge container holds its name, the minter's record that it has minted | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) | | the swarm controller's Grafana token | `swarm-bao-grafana-controller-token.service`, **on the host that runs Grafana**, as Grafana's built-in admin over its socket: it creates the `swarm-controller` service account as a Viewer (and sets it back to Viewer on every run), mints a token for it and writes it to the swarm secret store at `swarm/controller/swarm-controller/grafana/token` under Grafana's leaf. Re-run every ten minutes: it keeps a stored token Grafana accepts, and replaces one Grafana refuses with 401 (or a missing one), deleting every other token of that account first | the swarm secret store only; the controller reads it for `GET /api/alerts` and holds it in memory, re-reading the store once when Grafana answers 401 | nothing to place: Grafana's leaf is already required to run Grafana. Until the unit has run, `/api/alerts` answers 503 naming the empty path | | an agent's API-key environment (`ANTHROPIC_API_KEY`, `ANTHROPIC_BASE_URL`, an ACP provider key) | `hive-agent-backend-env.service`, in the agent's container, on an agent with a store that uses `useApiKey` or the `acp` runtime. When the store holds nothing at `swarm/agents//backend-env` and `backendEnvironmentFile` is a non-empty file, it creates the leaf with the file's exact bytes and `-cas=0`, under the agent's own leaf, which may create that path and not update it. It never rotates a value and never rewrites or deletes the file. To set or replace a key, use **set the API-key environment** in the swarm UI (`PUT /api/hives/{hive}/agents/{agent}/backend-env`): `swarm-controller` overwrites the leaf with its own grant, and the store's value then wins over the file. Its **remove** (`DELETE` on the same path) deletes the leaf, and an agent whose file has content then creates it from that file again | the swarm secret store at that path; the unit writes it to `/run/hive-agent-backend-env/env` (tmpfs, `0600`, root), and `hive-agent` and `hive-subagent-daemon` load that after `backendEnvironmentFile` as `EnvironmentFile=` | the same unit, same path. A variable in the store wins over the same one in the file. With the store down at a start, the file alone carries the agent, as it did before the store held a copy; with an older hive build the agent ignores the leaf and keeps reading the file | +| an agent's proxy settings (`HTTPS_PROXY`, `HTTP_PROXY`, `NO_PROXY`, the credential in the proxy URL) | `hive-agent-proxy-credential.service`, in the agent's container, as the agent user, at start and every 10 minutes: it reads `swarm/agents//proxy-credential` under the agent's own leaf and builds the file from it. It writes nothing while the path holds nothing or the hive's facts name no squid | `/run/hive-agent-proxy-credential/proxy.env` (tmpfs, `0600`, the agent user); the harness, its bash tool, the subagents, the MCP servers and the pollers load it as `EnvironmentFile=-` and read it when they start | With no credential, or on a hive that writes no `HIVE_SQUID_PROXY` and `HIVE_EGRESS_NO_PROXY` fact, nothing writes the file and a unit has no proxy variable and reaches the internet directly. A failed read keeps the file already in place. A unit started before the file appeared goes direct until its next start. The credential is also in the environment of every process those units start, so `env` in the agent's shell prints it | | an agent's claude login (`.credentials.json`) | `swarm-claude-login.service`, in its container, when an operator completes `claude auth login` from the agent's card in the swarm UI; it stores the file as a new version at `swarm/agents//claude-credentials` under its own leaf, which may create and update that path and read nothing under an agent | the swarm secret store at that path; the agent's `hive-agent` reads it under the agent's own leaf every 30 seconds while it waits for a login and writes it to `~/.claude/.credentials.json` | the same read, co-located or not; nothing for an operator to place beyond the agent's store leaf. When the store's copy and the agent's differ the KV version decides: the agent installs only a version newer than the last it installed (`claude-credentials-version` under the harness dir), because claude rotates the refresh token in its own copy and the stored one goes stale. The agent ignores a stale or equal version, and a relogin writes a newer one — [claude-login.md](claude-login.md) | | the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | | the queue container's journal forwarder's copy of the queue's OIDC secret | `swarm-journal-forwarder-oidc.service` inside the `swarm-nats` container reads it out of the swarm secret store under the responder's store leaf, and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. After a reboot the collector restarts until the file exists, then resumes from its saved journal cursor. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | @@ -156,7 +157,7 @@ pieces fit, and the reasoning behind them, is in [`credentials.md`](credentials. | the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/run/hive-forge-oidc/.secret` on the host (tmpfs), bound into the forge container, handed to `forgejo-sso-source` through `LoadCredential` | | the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` | | this hive's forge read token | `swarm-controller`, into the secret store at `swarm/hives//forge/token`, for the `hive-` account; a pass at start and every five minutes re-mints a missing or stale token | `hive-c0re` reads it under the hive's own store identity, and copies it (account on the first line, token on the second) to `/run/hyperhive/forge-hive-token` (tmpfs, `0600` `hive-core`) for nix's `git`, which `hive-priv` also runs. Each `internal/meta` read and each boot rewrites the copy when it differs; where they differ after a rotation, a build that starts before the next read fails to authenticate, and a retry reads the new token | -| squid's users file | `hive-c0re` (`workers::squid_users`), every 30 seconds, from the proxy credentials of its agents and the self-check | `services.hyperhive.squid.usersFile` (`/var/lib/hive-squid/users`, `hive-core:squid`, `0640`): `:` lines, SHA-512 crypt hashes only, renamed into place when it changed. `unrestricted` beside it lists the users whose agent holds the swarm's `unrestricted_internet` flag, names only; it lists every agent the hive never read as off `systemd-tmpfiles` creates it empty so squid's helper never sees a missing file. A hive whose agents have no credential yet (an older controller) has an empty file: every login fails and the self-check reports `ok=false`; no agent sends a credential, so no traffic changes | +| squid's users file | `hive-c0re` (`workers::squid_users`), every 30 seconds, from the proxy credentials of its agents and the self-check | `services.hyperhive.squid.usersFile` (`/var/lib/hive-squid/users`, `hive-core:squid`, `0640`): `:` lines, SHA-512 crypt hashes only, renamed into place when it changed. `unrestricted` beside it lists the users whose agent holds the swarm's `unrestricted_internet` flag, names only; it lists every agent the hive never read as off `systemd-tmpfiles` creates it empty so squid's helper never sees a missing file. A hive whose agents have no credential yet (an older controller) has an empty file: every login fails and the self-check reports `ok=false`; an agent without one sends none, so it goes direct | | the squid self-check's password | `swarm-controller` mints `swarm/hives//squid-selfcheck`; `squid-selfcheck-credential` (root, every 10 minutes) reads it as the hive | `/run/credstore/hive-squid.selfcheck` (tmpfs, root, `0400`), imported by `squid-selfcheck.service` as a systemd credential and handed to `curl` on its stdin. Both self-check users (`selfcheck.squid` and the always-unrestricted `selfcheck-unrestricted.squid`) use it, so whoever holds it has unrestricted egress through this squid Missing, or different from the users file: the self-check logs `credential=absent` or a 407 and fails, and nothing else breaks | ### Store identities, one per principal diff --git a/docs/swarm/ui.md b/docs/swarm/ui.md index d468c5b8..66676fd7 100644 --- a/docs/swarm/ui.md +++ b/docs/swarm/ui.md @@ -569,8 +569,9 @@ the swarm never set the flag, which counts as on. The **restrict** or permission list along with the flag, because a `PUT` without the list resets a stored list to the default. An agent with no stored list sends the flag alone. The hive's egress proxy applies the flag to each agent's proxy credential (see -[the egress proxy](../networking/network.md#egress-proxy-squid)); no agent sends -one yet, so the flag changes no traffic today. +[the egress proxy](../networking/network.md#egress-proxy-squid)). An agent sends its +credential from its next start after it has fetched it, so a flag change applies +to an agent's traffic from then on. A saved change isn't instant. The hive reads the caps about every 30 seconds, and a running agent takes a changed cap at its next spawn or rebuild, so the diff --git a/docs/trust-boundary/security.md b/docs/trust-boundary/security.md index e44ec02d..06e7b1c2 100644 --- a/docs/trust-boundary/security.md +++ b/docs/trust-boundary/security.md @@ -209,6 +209,11 @@ token policy bounds file reads; network isolation bounds network reach. nixbld users can't read it. The swarm-minted token lives under `/run/hive-agent-forge-token/` (`0700` directory, `0400` file), outside the state dir. +- `/run/hive-agent-proxy-credential/proxy.env` — the agent's egress-proxy + credential inside its proxy URL (`0700` directory, `0600` file, tmpfs). Units + load it as an environment file, so the credential is also in the environment of + every process they start. It authenticates this agent to the hive's squid and + nothing else. **Policy**: all credential files written to agent state directories MUST be mode `0600` or stricter. Don't create world-readable secret files in agent state dirs. diff --git a/hive-c0re/src/lifecycle/mod.rs b/hive-c0re/src/lifecycle/mod.rs index fbe481f1..beb68060 100644 --- a/hive-c0re/src/lifecycle/mod.rs +++ b/hive-c0re/src/lifecycle/mod.rs @@ -141,8 +141,8 @@ pub fn network_isolation_from_env() -> Result } /// The hive's facts every container is handed at runtime, so that no agent -/// build carries a value that differs between hosts. The name, domain, OTEL -/// resource labels and queue address come from hive-c0re's own unit environment +/// build carries a value that differs between hosts. The name, domain, egress +/// proxy and its direct hosts, OTEL resource labels and queue address come from hive-c0re's own unit environment /// (`nix/host-modules/hive-c0re/environment.nix`), where an unset or empty /// variable is `None`; the dashboard port and pinned claude from `hive`. fn hive_facts(hive: &HiveEnv) -> hive_priv_sock::HiveFacts { @@ -162,6 +162,8 @@ fn hive_facts_from( claude_code_path: hive.claude_code_path.clone(), otel_extra_resource_attributes: read("HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES"), agent_nats_url: read("HIVE_AGENT_NATS_URL"), + squid_proxy: read("HIVE_SQUID_PROXY"), + egress_no_proxy: read("HIVE_EGRESS_NO_PROXY"), } } diff --git a/hive-c0re/src/lifecycle/tests.rs b/hive-c0re/src/lifecycle/tests.rs index d638cdb2..9a7ec33a 100644 --- a/hive-c0re/src/lifecycle/tests.rs +++ b/hive-c0re/src/lifecycle/tests.rs @@ -541,6 +541,8 @@ fn hive_facts_carry_the_per_host_values() { let facts = hive_facts_from(&hive, |name| match name { "HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES" => Some("team=a".to_owned()), "HIVE_AGENT_NATS_URL" => Some("tls://nats.t.local:4222".to_owned()), + "HIVE_SQUID_PROXY" => Some("10.42.0.1:3128".to_owned()), + "HIVE_EGRESS_NO_PROXY" => Some("10.42.0.1,t.local".to_owned()), _ => None, }); assert_eq!(facts.dashboard_port, Some(7123)); @@ -556,6 +558,8 @@ fn hive_facts_carry_the_per_host_values() { facts.agent_nats_url.as_deref(), Some("tls://nats.t.local:4222") ); + assert_eq!(facts.squid_proxy.as_deref(), Some("10.42.0.1:3128")); + assert_eq!(facts.egress_no_proxy.as_deref(), Some("10.42.0.1,t.local")); } /// Unset and empty both leave a fact out, so the agent falls back to its @@ -568,5 +572,7 @@ fn hive_facts_leave_unset_values_out() { assert_eq!(facts.claude_code_path, None); assert_eq!(facts.otel_extra_resource_attributes, None); assert_eq!(facts.agent_nats_url, None); + assert_eq!(facts.squid_proxy, None); + assert_eq!(facts.egress_no_proxy, None); assert_eq!(facts.name, None); } diff --git a/hive-priv-sock/src/lib.rs b/hive-priv-sock/src/lib.rs index 4b02dfc7..9524b49f 100644 --- a/hive-priv-sock/src/lib.rs +++ b/hive-priv-sock/src/lib.rs @@ -243,6 +243,14 @@ pub struct HiveFacts { /// `HIVE_AGENT_NATS_URL`: where the agent reaches the swarm queue, from /// `deploy.hive-controller.queue.agentNatsUrl`. pub agent_nats_url: Option, + /// `HIVE_SQUID_PROXY`: `:` of this host's squid, which + /// checks each agent's proxy credential and applies the agent's + /// `unrestricted_internet` flag. Set only by a hive that carries that + /// policy; an agent sends nothing through a proxy without it. + pub squid_proxy: Option, + /// `HIVE_EGRESS_NO_PROXY`: the comma-separated hosts an agent must reach + /// directly and never through the squid above. + pub egress_no_proxy: Option, } /// Wall-clock bound hive-priv puts on one [`PrivRequest::MeasureAgentDisk`] diff --git a/hive-priv/src/container_config.rs b/hive-priv/src/container_config.rs index d9281702..6b79966e 100644 --- a/hive-priv/src/container_config.rs +++ b/hive-priv/src/container_config.rs @@ -409,6 +409,20 @@ fn render_hive_facts(facts: &HiveFacts) -> Result { 253, &|b: u8| dns(b) || matches!(b, b':' | b'/'), ), + // `:`. + ( + "HIVE_SQUID_PROXY", + facts.squid_proxy.as_ref(), + 21, + &|b: u8| b.is_ascii_digit() || matches!(b, b'.' | b':'), + ), + // `host,host`: bare names and addresses. + ( + "HIVE_EGRESS_NO_PROXY", + facts.egress_no_proxy.as_ref(), + 4096, + &|b: u8| dns(b) || b == b',', + ), ] { let Some(value) = value else { continue }; if value.is_empty() || value.len() > max_len || !value.bytes().all(allowed) { @@ -803,6 +817,59 @@ mod tests { ); } + /// The egress facts: the proxy as `:`, and the hosts that skip it. + #[test] + fn hive_facts_render_the_egress_proxy() { + let facts = HiveFacts { + squid_proxy: Some("10.42.0.1:3128".to_owned()), + egress_no_proxy: Some("10.42.0.1,pr1ma.example.test,forge.example.test".to_owned()), + ..HiveFacts::default() + }; + assert_eq!( + render_hive_facts(&facts).unwrap(), + "HIVE_SQUID_PROXY=10.42.0.1:3128\n\ + HIVE_EGRESS_NO_PROXY=10.42.0.1,pr1ma.example.test,forge.example.test\n" + ); + } + + /// Neither egress fact may carry a second assignment, a space, a quote or a + /// URL: the agent builds its proxy URL and `NO_PROXY` from them. + #[test] + fn hive_facts_refuse_bad_egress_values() { + for bad in [ + "", + "10.42.0.1:3128\nHIVE_AGENT_NATS_URL=x", + "http://10.42.0.1:3128", + "10.42.0.1:3128 ", + "u:p@10.42.0.1:3128", + ] { + let facts = HiveFacts { + squid_proxy: Some(bad.to_owned()), + ..HiveFacts::default() + }; + assert!( + render_hive_facts(&facts).is_err(), + "{bad:?} must be refused" + ); + } + for bad in [ + "", + "a.test,b.test\nPATH=/x", + "a.test b.test", + "\"a.test\"", + "$(x)", + ] { + let facts = HiveFacts { + egress_no_proxy: Some(bad.to_owned()), + ..HiveFacts::default() + }; + assert!( + render_hive_facts(&facts).is_err(), + "{bad:?} must be refused" + ); + } + } + /// The claude path is a top-level store path or nothing, and neither /// value may carry generator syntax. #[test] diff --git a/nix/agent-modules/default.nix b/nix/agent-modules/default.nix index ca73c8d7..9682b8b8 100644 --- a/nix/agent-modules/default.nix +++ b/nix/agent-modules/default.nix @@ -71,6 +71,7 @@ in ./claude-settings.nix ./dashboard-links.nix ./docs.nix + ./egress-proxy.nix ./forge.nix ./forge-accounts.nix ./forge-token.nix diff --git a/nix/agent-modules/egress-proxy.nix b/nix/agent-modules/egress-proxy.nix new file mode 100644 index 00000000..d40d485d --- /dev/null +++ b/nix/agent-modules/egress-proxy.nix @@ -0,0 +1,240 @@ +# This agent's own egress-proxy credential, fetched from the swarm secret store +# by the agent itself, and the proxy environment built from it. +# +# `swarm-controller` mints the credential at +# `swarm/agents//proxy-credential` +# (`swarm_secret_client::proxy::agent_credential_path`); the hive's squid checks +# it against the hash `hive-c0re` renders. This unit logs in with the +# certificate ./bao.nix keeps current, reads the agent's own path, and writes +# `proxy.env` for the units below to load as an `EnvironmentFile=`: +# +# - `HTTPS_PROXY` and `HTTP_PROXY`, upper and lower case, with the credential in +# the URL's userinfo, which Claude Code, Node, curl, git and reqwest send as +# `Proxy-Authorization`; +# - `NODE_USE_ENV_PROXY=1`, without which Node's global `fetch` ignores them; +# - `NO_PROXY` and `no_proxy`. +# +# The file is written only when the credential AND the hive's facts are both +# there. `HIVE_SQUID_PROXY` (the proxy's address) and `HIVE_EGRESS_NO_PROXY` (the +# hosts that skip it) are written by a hive whose squid checks credentials and +# applies each agent's flag; on any other hive the agent has no proxy variable +# and reaches the internet directly. The units load the file with a leading `-`, +# so an agent without it is unchanged. A failed read keeps the file already in +# place, and units read it when they start, so an agent takes the proxy at its +# next start. +# +# `NO_PROXY` is the loopback names and the hive's list as it is: squid refuses +# every destination that resolves to the bridge, and the hive's resolver answers +# the bridge address for the swarm's services. Node has no CIDR support, so the +# list holds the bridge as an address. +{ + pkgs, + lib, + config, + ... +}: +let + cfg = config.services.hyperhive.agent.bao; + agentCfg = config.services.hyperhive.agent; + + # The name `swarm-controller` minted the credential under — see ./queue-identity.nix. + agentName = agentCfg.user.name; + + # The server CA id ./bao.nix loads. + serverCaCredential = "hive-agent-bao-server-ca"; + + unitName = "hive-agent-proxy-credential"; + + # The nix half of `swarm_secret_client::proxy::agent_credential_path` plus + # `path::MOUNT`. + credentialPath = "secret/swarm/agents/${agentName}/proxy-credential"; + + runtimeDir = unitName; + envFile = "/run/${runtimeDir}/proxy.env"; + # Beside the file, so the rename that replaces it stays in one directory. + stagingFile = "/run/${runtimeDir}/proxy.env.new"; + secretFile = "/run/${runtimeDir}/secret.new"; + # bao's stderr, in the unit's own `0700` directory rather than `/tmp`. + errFile = "/run/${runtimeDir}/bao.err"; + + # Written by `hive-priv` into every container (./hive-facts.nix). The two + # egress facts are read here and nowhere else: the generator there exports + # only the keys it names, so neither reaches a unit's environment. + factsFile = "/etc/hyperhive-hive-facts"; + + loopback = [ + "localhost" + "127.0.0.1" + "::1" + ]; + + # The store's address is the whole switch, as in ./bao.nix and + # ./forge-token.nix. + configured = cfg.addr != null; + + storeRetry = import ../host-modules/lib/store-retry.nix { }; + + # Units that make outbound calls: the harness and the claude it starts, the + # bash tool and the subagents it runs, MCP servers, and the pollers. They + # exist under the same conditions as their definitions, because a unit that + # is not defined would be generated inert. + consumers = [ + "hive-agent" + "hive-bash-daemon" + "hive-subagent-daemon" + "hive-mcp-http" + "hive-forge-notify" + ] + ++ lib.optional (agentCfg.matrixAccounts != { }) "hive-matrix-daemon" + ++ lib.optional agentCfg.github.enable "hive-github-notify" + ++ lib.optional (agentCfg.icon != null && agentCfg.forge.url != null) "forge-avatar-sync"; +in +{ + config = lib.mkIf configured { + systemd.services = { + ${unitName} = { + description = "fetch this agent's egress-proxy credential from the secret store"; + after = [ + "network.target" + # Ordering only: that unit reports a store this agent cannot reach as + # itself. + "hive-agent-bao-identity.service" + ]; + wantedBy = [ "multi-user.target" ]; + path = [ + pkgs.openbao + pkgs.coreutils + pkgs.diffutils + ]; + # ../host-modules/lib/store-retry.nix. + inherit (storeRetry) startLimitBurst startLimitIntervalSec; + serviceConfig = storeRetry.serviceConfig // { + Type = "oneshot"; + # Not `RemainAfterExit`: the timer below has to be able to start this + # unit again. `RuntimeDirectoryPreserve` keeps the file between runs. + RemainAfterExit = false; + TimeoutStartSec = 30; + User = agentName; + Group = agentName; + RuntimeDirectory = runtimeDir; + RuntimeDirectoryMode = "0700"; + RuntimeDirectoryPreserve = "yes"; + UMask = "0177"; + LoadCredential = [ serverCaCredential ]; + }; + environment = { + BAO_ADDR = cfg.addr; + BAO_CLIENT_CERT = cfg.clientCertFile; + BAO_CLIENT_KEY = cfg.clientKeyFile; + }; + script = '' + set -euo pipefail + + # No identity delivered: ./bao.nix's check reports that. + for f in "$BAO_CLIENT_CERT" "$BAO_CLIENT_KEY"; do + if [ ! -s "$f" ]; then + echo "this agent has no store identity, so it cannot fetch its egress-proxy credential." >&2 + exit 0 + fi + if [ ! -r "$f" ]; then + echo "cannot read $f, so this agent cannot present its store identity." >&2 + exit 1 + fi + done + + # The hive says its squid checks each agent's credential and applies + # its flag by writing both facts. Without them (a hive that predates + # them) there is no proxy to name, and the store is not asked. + proxy= direct= + if [ -r ${factsFile} ]; then + while IFS='=' read -r key value || [ -n "$key" ]; do + case "$key" in + HIVE_SQUID_PROXY) proxy=$value ;; + HIVE_EGRESS_NO_PROXY) direct=$value ;; + esac + done < ${factsFile} + fi + if [[ ! "$proxy" =~ ^[0-9]+(\.[0-9]+){3}:[0-9]+$ || ! "$direct" =~ ^[A-Za-z0-9.,_-]+$ ]]; then + echo "this hive's facts name no squid proxy, so this agent stays direct." >&2 + exit 0 + fi + + if [ -s "$CREDENTIALS_DIRECTORY/${serverCaCredential}" ]; then + export BAO_CACERT="$CREDENTIALS_DIRECTORY/${serverCaCredential}" + fi + + err=${lib.escapeShellArg errFile} + trap 'rm -f "$err" ${lib.escapeShellArg stagingFile} ${lib.escapeShellArg secretFile}' EXIT + + # 🩸 Every exit below that is not a rename leaves ${envFile} as it + # was: a store that is briefly unreachable, or has no credential for + # this agent yet, must not take a working proxy away, and must not + # invent one. + rm -f "$err" + if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then + echo "could not log in to the swarm secret store at $BAO_ADDR; the proxy settings in place, if any, are kept:" >&2 + if [ -s "$err" ]; then cat "$err" >&2; fi + exit 1 + fi + export BAO_TOKEN + + # ⚠️ Written by redirect, never echoed: the field is the secret. + rm -f "$err" ${lib.escapeShellArg secretFile} + if ! bao kv get -field=value ${lib.escapeShellArg credentialPath} > ${lib.escapeShellArg secretFile} 2>"$err"; then + echo "no egress-proxy credential at ${credentialPath} yet; this agent sends no proxy variables until there is one." >&2 + if [ -s "$err" ]; then cat "$err" >&2; fi + exit 0 + fi + secret="$(<${lib.escapeShellArg secretFile})" + + # The controller mints base64url, which needs no escaping in a URL's + # userinfo or an env file. Anything else is not that secret. + if [[ ! "$secret" =~ ^[A-Za-z0-9_-]+$ ]]; then + echo "the value at ${credentialPath} is not a base64url secret, so no proxy variables are written." >&2 + exit 1 + fi + + no_proxy=${lib.escapeShellArg (lib.concatStringsSep "," loopback)},$direct + url="http://${agentName}:$secret@$proxy" + + rm -f ${lib.escapeShellArg stagingFile} + { + printf 'HTTPS_PROXY=%s\n' "$url" + printf 'HTTP_PROXY=%s\n' "$url" + printf 'https_proxy=%s\n' "$url" + printf 'http_proxy=%s\n' "$url" + printf 'NODE_USE_ENV_PROXY=1\n' + printf 'NO_PROXY=%s\n' "$no_proxy" + printf 'no_proxy=%s\n' "$no_proxy" + } > ${lib.escapeShellArg stagingFile} + + if cmp -s ${lib.escapeShellArg stagingFile} ${lib.escapeShellArg envFile}; then + echo "this agent's proxy settings are unchanged." + exit 0 + fi + mv -f ${lib.escapeShellArg stagingFile} ${lib.escapeShellArg envFile} + echo "wrote this agent's proxy settings from ${credentialPath}." + ''; + }; + } + // lib.genAttrs consumers (_: { + # `wants`, not `requires`: a store outage must not stop the harness, which + # then starts direct. `after` waits for one attempt. + wants = [ "${unitName}.service" ]; + after = [ "${unitName}.service" ]; + # A list, which merges with the entries other modules give a unit. + serviceConfig.EnvironmentFile = [ "-${envFile}" ]; + }); + + # The controller re-checks every agent's credential every five minutes; the + # fetch picks one up within about ten more. + systemd.timers.${unitName} = { + description = "re-fetch this agent's egress-proxy credential from the secret store"; + wantedBy = [ "timers.target" ]; + timerConfig = { + OnUnitInactiveSec = "10min"; + RandomizedDelaySec = "1min"; + }; + }; + }; +} diff --git a/nix/agent-modules/hive-facts.nix b/nix/agent-modules/hive-facts.nix index e9010ebf..db6dc1b7 100644 --- a/nix/agent-modules/hive-facts.nix +++ b/nix/agent-modules/hive-facts.nix @@ -12,6 +12,8 @@ # `OTEL_RESOURCE_ATTRIBUTES` by ./otel.nix (no extra labels when absent). # - `HIVE_AGENT_NATS_URL`, where the harness reaches the swarm queue (it fails # at boot when absent). +# - `HIVE_SQUID_PROXY` and `HIVE_EGRESS_NO_PROXY`: the squid address and the hosts +# that skip it, read only by ./egress-proxy.nix; the generator drops them. # # Any of them may be absent: a hive-c0re older than the agent's build writes # fewer. Two readers hand the lines on: diff --git a/nix/checks.nix b/nix/checks.nix index 1d194d25..6ff29e1d 100644 --- a/nix/checks.nix +++ b/nix/checks.nix @@ -183,7 +183,8 @@ in inherit (pkgs) lib; }; # Executes what the two `module-eval-agent-*-bao` suites above only - # evaluate: both fetch units' rendered scripts, against a stub `bao`. + # evaluate: the rendered scripts of the forge-token, identity and + # proxy-credential fetch units, against a stub `bao`. script-test-agent-bao-fetch = import ./script-tests/agent-bao-fetch.nix { inherit pkgs self nixosSystem; inherit (pkgs) lib; diff --git a/nix/host-modules/hive-c0re/environment.nix b/nix/host-modules/hive-c0re/environment.nix index d342f9fb..867245ef 100644 --- a/nix/host-modules/hive-c0re/environment.nix +++ b/nix/host-modules/hive-c0re/environment.nix @@ -15,6 +15,30 @@ let # sharing a host with the store. Deriving this from # `deploy.bao.enable` would be the co-location assumption itself. haveBaoClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null; + + # What an agent reaches directly and never through squid. Squid refuses every + # destination that resolves to the bridge, and the hive's resolver answers the + # bridge address for the gateway's names, so each of these must skip it. A bare + # name matches itself and its subdomains in curl, Node and reqwest. + egressDirectHosts = lib.unique ( + lib.filter (n: n != null && n != "") ( + [ + # Node's NO_PROXY has no CIDR support, so the bridge is listed as the address. + config.services.hyperhive.network.bridgeIp + # The hive's own zone, which dnsmasq answers with the bridge IP. + config.services.hyperhive.domain + # The mTLS store and the queue: not in `serviceDomains`. + config.services.hyperhive.swarm.bao.domain + config.services.hyperhive.swarm.nats.domain + ] + # The swarm's apex (the swarm UI and the controller) and its forge, matrix, + # attic, logs, metrics and otel services, wherever they sit. The store and + # queue names are not in it, hence their lines above. + ++ config.services.hyperhive.swarm.serviceDomains + # Every name this host's gateway answers with the bridge IP. + ++ config.services.hyperhive.gateway.localNames + ) + ); in { # nix (the prebuild `nix build`, flake-check, and meta eval) writes @@ -277,6 +301,13 @@ in # store under the identity above. HIVE_SQUID_USERS_FILE = config.services.hyperhive.squid.usersFile; HIVE_SQUID_UNRESTRICTED_FILE = config.services.hyperhive.squid.unrestrictedFile; + # The egress proxy facts every container is handed (`HiveFacts` in + # `hive-priv-sock`). They sit with the users file above because that is what + # makes squid's credential check and per-agent policy real on this host: an + # agent sends nothing through the proxy unless it reads both. Absent on a + # hive that predates them, so its agents go direct. + HIVE_SQUID_PROXY = config.services.hyperhive.squid.listen; + HIVE_EGRESS_NO_PROXY = lib.concatStringsSep "," egressDirectHosts; } // lib.optionalAttrs (haveBaoClientIdentity && baoDeploy.serverCaFile != null) { # Absent means the system trust store — right for a deployment with a real diff --git a/nix/host-modules/hive-squid.nix b/nix/host-modules/hive-squid.nix index 7f1e107e..0875d9ee 100644 --- a/nix/host-modules/hive-squid.nix +++ b/nix/host-modules/hive-squid.nix @@ -4,9 +4,9 @@ # must carry a per-agent proxy credential, checked against the users file # hive-c0re renders. A credential's agent either has the swarm's # `unrestricted_internet` flag, and may then reach any destination the deny -# rules leave, or reaches only the listed names. No agent sends a credential -# yet; the exporter, journal forwarding and the self-check below are how a -# deploy shows it works. +# rules leave, or reaches only the listed names. An agent sends its credential +# once it has fetched it (../agent-modules/egress-proxy.nix); the exporter, +# journal forwarding and the self-check below show the proxy works. { pkgs, lib, @@ -19,6 +19,7 @@ let netCfg = hyperhiveCfg.network; port = 3128; + listen = "${netCfg.bridgeIp}:${toString port}"; proxyUrl = "http://${netCfg.bridgeIp}:${toString port}"; exporterListen = "127.0.0.1:9301"; denyCanary = "deny-canary.invalid"; @@ -67,7 +68,7 @@ let # port, and every other user to the listed names. The `dst` denies cost an # unlisted name a lookup on the host's resolver before it is refused. squidConf = '' - http_port ${netCfg.bridgeIp}:${toString port} + http_port ${listen} pid_filename /run/squid/squid.pid cache_log stdio:/var/log/squid/cache.log coredump_dir /var/cache/squid @@ -197,6 +198,19 @@ let in { options.services.hyperhive.squid = { + listen = lib.mkOption { + type = lib.types.str; + default = listen; + readOnly = true; + defaultText = lib.literalMD "`:3128`"; + description = '' + The address squid listens on, as `:`. Read-only. + hive-c0re hands it to every container as the `HIVE_SQUID_PROXY` fact, + which is what tells an agent that this host's squid checks its proxy + credential and applies its `unrestricted_internet` flag. + ''; + }; + allowedDomains = lib.mkOption { type = lib.types.listOf lib.types.str; default = hyperhiveCfg.swarm.serviceDomains; diff --git a/nix/module-eval/agent-forge-bao.nix b/nix/module-eval/agent-forge-bao.nix index 8f51ebc8..04fe2cb0 100644 --- a/nix/module-eval/agent-forge-bao.nix +++ b/nix/module-eval/agent-forge-bao.nix @@ -3,7 +3,9 @@ # not executes"). # # The agent side of the swarm-minted forge token: ../agent-modules/forge-token.nix -# fetches it, and ../agent-modules/forge.nix's readers find it. +# fetches it, and ../agent-modules/forge.nix's readers find it. The same fixtures +# cover ../agent-modules/egress-proxy.nix, the agent's proxy credential and the +# environment built from it. { pkgs, lib, @@ -31,6 +33,8 @@ let agentForgeBao = agentWith { services.hyperhive.agent.bao.addr = baoAddr; services.hyperhive.agent.forge.url = forgeUrl; + # A second `EnvironmentFile=` on the harness, which the proxy file must join. + services.hyperhive.agent.backendEnvironmentFile = "/var/lib/t/backend.env"; services.hyperhive.agent.icon = pkgs.emptyFile; }; @@ -42,8 +46,22 @@ let }; fetchUnit = machine: machine.systemd.services.hive-agent-forge-token; + proxyUnit = machine: machine.systemd.services.hive-agent-proxy-credential; accountsUnit = machine: machine.systemd.services.hive-agent-forge-accounts; tokenFile = machine: machine.services.hyperhive.agent.forge.tokenFile; + + proxyEnvEntry = "-/run/hive-agent-proxy-credential/proxy.env"; + # The units that make outbound calls and so load the proxy file. This fixture + # has an icon, a forge and GitHub on, and no matrix accounts. + proxyConsumers = [ + "hive-agent" + "hive-bash-daemon" + "hive-subagent-daemon" + "hive-mcp-http" + "hive-forge-notify" + "hive-github-notify" + "forge-avatar-sync" + ]; in let cases = [ @@ -238,6 +256,137 @@ let name = "the account fetch re-runs every two minutes"; ok = agentForgeBao.systemd.timers.hive-agent-forge-accounts.timerConfig.OnUnitInactiveSec == "2min"; } + { + # ../agent-modules/egress-proxy.nix. Presence arm: the unit and its timer + # exist, and the agent is told nothing until a credential is fetched. + name = "an agent with a store address fetches its egress-proxy credential, and one without does not"; + ok = + agentForgeBao.systemd.services ? hive-agent-proxy-credential + && agentForgeBao.systemd.timers ? hive-agent-proxy-credential + && !(agentForgeNoBao.systemd.services ? hive-agent-proxy-credential) + && !(agentForgeNoBao.systemd.timers ? hive-agent-proxy-credential); + } + { + # The nix half of `swarm_secret_client::proxy::agent_credential_path` + # plus the mount. + name = "the proxy fetch reads the agent's own proxy-credential path, by redirect, and never echoes it"; + ok = + let + name = agentForgeBao.services.hyperhive.agent.user.name; + s = (proxyUnit agentForgeBao).script; + in + lib.hasInfix "secret/swarm/agents/${name}/proxy-credential" s + && lib.hasInfix "-field=value" s + && lib.hasInfix "> ${lib.escapeShellArg "/run/hive-agent-proxy-credential/secret.new"}" s + && !(lib.hasInfix "echo \"$(bao" s) + && !(lib.hasInfix "echo $(bao" s) + && !(lib.hasInfix "echo \"$secret" s) + && !(lib.hasInfix "printf '%s\\n' \"$secret" s); + } + { + name = "the proxy fetch runs as the agent with its own store identity and keeps its file between runs"; + ok = + let + u = proxyUnit agentForgeBao; + c = u.serviceConfig; + in + c.User == agentForgeBao.services.hyperhive.agent.user.name + && c.UMask == "0177" + && !c.RemainAfterExit + && c.RuntimeDirectory == "hive-agent-proxy-credential" + && c.RuntimeDirectoryMode == "0700" + && c.RuntimeDirectoryPreserve == "yes" + && u.environment.BAO_ADDR == baoAddr + && u.environment.BAO_CLIENT_CERT == "/run/hive-agent-bao-identity/current/cert"; + } + { + # Presence: every outbound unit loads the file, and the harness keeps the + # entry it already had. Absence: an agent with no store loads none. + name = "every outbound unit loads the proxy file optionally, beside its other environment files"; + ok = + let + svc = agentForgeBao.systemd.services; + files = c: svc.${c}.serviceConfig.EnvironmentFile or [ ]; + in + lib.all ( + c: + lib.elem proxyEnvEntry (files c) + && lib.elem "hive-agent-proxy-credential.service" svc.${c}.after + && lib.elem "hive-agent-proxy-credential.service" svc.${c}.wants + && !(lib.elem "hive-agent-proxy-credential.service" (svc.${c}.requires or [ ])) + ) proxyConsumers + && lib.elem "-/var/lib/t/backend.env" (files "hive-agent") + && !(lib.any + ( + c: + lib.elem proxyEnvEntry (agentForgeNoBao.systemd.services.${c}.serviceConfig.EnvironmentFile or [ ]) + ) + [ + "hive-agent" + "hive-bash-daemon" + "hive-forge-notify" + ] + ); + } + { + # The credential is in a file only the fetch writes. Nothing static (a + # unit environment, the global one, the login shells) carries a proxy + # variable, so an agent without a fetched credential goes direct. + name = "no proxy variable is set where the credential is not"; + ok = + let + names = [ + "HTTPS_PROXY" + "HTTP_PROXY" + "https_proxy" + "http_proxy" + "NO_PROXY" + "no_proxy" + "NODE_USE_ENV_PROXY" + ]; + anywhere = + m: n: + m.systemd.globalEnvironment ? ${n} + || m.environment.variables ? ${n} + || lib.any (s: s.environment ? ${n}) (lib.attrValues m.systemd.services); + in + !(lib.any (anywhere agentForgeBao) names) && !(lib.any (anywhere agentForgeNoBao) names); + } + { + # The hive's facts are the only source of the proxy address and of the + # hosts that skip it, and they are read before the store is asked: a hive + # without them gets no login and no file. The loopback names stay local. + name = "the proxy file takes the proxy and the direct hosts from the hive's facts, before it asks the store"; + ok = + let + s = (proxyUnit agentForgeBao).script; + at = needle: lib.stringLength (lib.head (lib.splitString needle s)); + in + lib.hasInfix "HIVE_SQUID_PROXY) proxy=$value" s + && lib.hasInfix "HIVE_EGRESS_NO_PROXY) direct=$value" s + && lib.hasInfix "no_proxy=localhost,127.0.0.1,::1,$direct" s + && lib.hasInfix "/etc/hyperhive-hive-facts" s + && at "HIVE_SQUID_PROXY)" < at "bao login" + && !(lib.hasInfix "hyperhive-bridge-dns" s); + } + { + name = "the proxy file names every variable the clients read, and the credential only in the URL's userinfo"; + ok = + let + s = (proxyUnit agentForgeBao).script; + name = agentForgeBao.services.hyperhive.agent.user.name; + in + lib.all (v: lib.hasInfix "printf '${v}=%s\\n'" s) [ + "HTTPS_PROXY" + "HTTP_PROXY" + "https_proxy" + "http_proxy" + "NO_PROXY" + "no_proxy" + ] + && lib.hasInfix "NODE_USE_ENV_PROXY=1" s + && lib.hasInfix "url=\"http://${name}:$secret@$proxy\"" s; + } ]; in runGroup "agent-forge-bao" cases diff --git a/nix/module-eval/hive-otel.nix b/nix/module-eval/hive-otel.nix index 94294594..6bf1e61f 100644 --- a/nix/module-eval/hive-otel.nix +++ b/nix/module-eval/hive-otel.nix @@ -622,6 +622,68 @@ let builtins.elem 3128 fw.interfaces.${hiveOtel.services.hyperhive.network.bridgeName}.allowedTCPPorts && !(builtins.elem 3128 fw.allowedTCPPorts); } + { + # The facts that tell an agent this host's squid checks its credential and + # applies its flag. Presence arm: a hive holding a store identity writes + # both, and the direct hosts carry the names the brief's destinations + # resolve under. Absence arm: a hive with no identity renders no users + # file, so it announces no proxy. + name = "hive-c0re hands agents the squid address and the hosts that skip it, only with a store identity"; + ok = + let + e = hiveOtelAllLocal.systemd.services.hive-c0re.environment; + s = hiveOtelAllLocal.services.hyperhive; + hosts = lib.splitString "," e.HIVE_EGRESS_NO_PROXY; + in + e.HIVE_SQUID_PROXY == "${s.network.bridgeIp}:3128" + && e.HIVE_SQUID_PROXY == s.squid.listen + && lib.all (h: lib.elem h hosts) ( + [ + s.network.bridgeIp + s.swarm.domain + s.swarm.bao.domain + s.swarm.nats.domain + s.swarm.attic.domain + s.swarm.forge.domain + s.swarm.matrix.gatewayHost + s.swarm.victorialogs.domain + s.swarm.victoriametrics.domain + s.swarm.otel.domain + s.otel.collector.host + s.domain + ] + ++ s.gateway.localNames + ) + && !(lib.elem "example.com" hosts) + && lib.length hosts == lib.length (lib.unique hosts) + && !(hiveOtelBare.systemd.services.hive-c0re.environment ? HIVE_SQUID_PROXY) + && !(hiveOtelBare.systemd.services.hive-c0re.environment ? HIVE_EGRESS_NO_PROXY); + } + { + # The store and the queue are in no list the gateway builds, so a hive whose + # store is elsewhere gets them from their own options. Control: this + # fixture runs no store, so the gateway does not name it. + name = "a hive without a local store still lists the store and the queue as direct hosts"; + ok = + let + s = hiveOtelLeaf.services.hyperhive; + hosts = lib.splitString "," hiveOtelLeaf.systemd.services.hive-c0re.environment.HIVE_EGRESS_NO_PROXY; + in + !(lib.elem s.swarm.bao.domain s.gateway.localNames) + && lib.elem s.swarm.bao.domain hosts + && lib.elem s.swarm.nats.domain hosts; + } + { + # Squid's listener and the fact are one value. + name = "the squid fact is the address squid binds"; + ok = + let + s = hiveOtelAllLocal.services.hyperhive; + in + lib.elem "http_port ${s.squid.listen}" ( + lib.splitString "\n" hiveOtelAllLocal.services.squid.configText + ); + } ]; in runGroup "hive-otel" cases diff --git a/nix/script-tests/agent-bao-fetch.nix b/nix/script-tests/agent-bao-fetch.nix index 8e4a9312..65d7901e 100644 --- a/nix/script-tests/agent-bao-fetch.nix +++ b/nix/script-tests/agent-bao-fetch.nix @@ -1,7 +1,7 @@ # `checks.script-test-agent-bao-fetch` — runs the agent units that log in to -# the swarm secret store and fetch a secret, ../agent-modules/forge-token.nix -# and ../agent-modules/bao.nix, against a stub `bao`. The cases are in -# ./agent-bao-fetch.sh. +# the swarm secret store and fetch a secret, ../agent-modules/forge-token.nix, +# ../agent-modules/bao.nix and ../agent-modules/egress-proxy.nix, against a stub +# `bao`. The cases are in ./agent-bao-fetch.sh. # # What runs is the unit's rendered `ExecStart`, on the unit's own `PATH` with # the stub in front. The one edit is the unit's `/run//` prefix, moved @@ -77,6 +77,7 @@ pkgs.runCommand "hyperhive-script-test-agent-bao-fetch" ( unitEnv "FORGE" "hive-agent-forge-token" // unitEnv "IDENT" "hive-agent-bao-identity" + // unitEnv "PROXY" "hive-agent-proxy-credential" // { FAKE_BAO_BIN = "${fakeBao}/bin"; } diff --git a/nix/script-tests/agent-bao-fetch.sh b/nix/script-tests/agent-bao-fetch.sh index 74ad8476..4082e71c 100644 --- a/nix/script-tests/agent-bao-fetch.sh +++ b/nix/script-tests/agent-bao-fetch.sh @@ -1,5 +1,5 @@ # Cases for ./agent-bao-fetch.nix. Each case gets a fresh runtime directory and -# credentials directory, runs one unit's script under the unit's `UMask=0377` +# credentials directory, runs one unit's script under the unit's own `UMask=` # with a clean environment, and asserts on the exit code, the output, the # files left behind and the `bao` calls made. set -uo pipefail @@ -48,7 +48,13 @@ setup() { client_cert=$creds/hive-agent-bao-cert client_key=$creds/hive-agent-bao-key fi : >"$log" - if ! sed "s|/run/$unit/|$rt/|g" "${!script_var}" >"$dir/script"; then + # The proxy fetch reads the egress facts the hive writes into a + # container, moved under the build directory like /run. + mkdir "$dir/etc" + printf 'HIVE_DASHBOARD_PORT=7000\nHIVE_SQUID_PROXY=10.42.0.1:3128\nHIVE_EGRESS_NO_PROXY=10.42.0.1,t.hive,bao.t.local\n' >"$dir/etc/hyperhive-hive-facts" + umask_value=0377 + if [ "$prefix" = PROXY ]; then umask_value=0177; fi + if ! sed -e "s|/run/$unit/|$rt/|g" -e "s|/etc/hyperhive-|$dir/etc/hyperhive-|g" "${!script_var}" >"$dir/script"; then echo "cannot read the rendered script for $unit" >&2 exit 1 fi @@ -68,7 +74,7 @@ setup() { go() { count=$((count + 1)) ( - umask 0377 + umask "$umask_value" cd "$dir" || exit 99 exec env -i \ PATH="$dir/bin:$FAKE_BAO_BIN:$unit_path" \ @@ -382,6 +388,177 @@ for n in 1 2 3; do expect_pair cert2 key2 done +# The proxy fetch writes the agent's proxy variables, credential included, to +# one file the units load as an `EnvironmentFile=`. Nothing here may print the +# credential, so every case checks the output for it. +proxy_url=http://a1:the-secret@10.42.0.1:3128 +proxy_no=localhost,127.0.0.1,::1,10.42.0.1,t.hive,bao.t.local +proxy_path=secret/swarm/agents/a1/proxy-credential +proxy_kv_call="kv get -field=value $proxy_path" +proxy_env() { + printf 'HTTPS_PROXY=%s\nHTTP_PROXY=%s\nhttps_proxy=%s\nhttp_proxy=%s\nNODE_USE_ENV_PROXY=1\nNO_PROXY=%s\nno_proxy=%s\n' \ + "$proxy_url" "$proxy_url" "$proxy_url" "$proxy_url" "$1" "$1" +} + +expect_secret_unprinted() { + if grep -qF -e the-secret -e s.fake-token "$dir/out" "$dir/err"; then + fail "a secret reached the unit's output: $(cat "$dir/out" "$dir/err")" + fi +} + +setup PROXY "no store identity delivered" +: >"$creds/hive-agent-bao-cert" +go +expect_rc 0 +expect_err "has no store identity" +expect_calls +expect_no_file "$rt/proxy.env" + +setup PROXY "a first fetch writes the variables 0600, and prints no secret" +go +expect_rc 0 +expect_out "wrote this agent's proxy settings from $proxy_path" +expect_calls "$login_call" "$proxy_kv_call" +expect_file "$rt/proxy.env" "$(proxy_env "$proxy_no")" +if [ "$(stat -c %a "$rt/proxy.env")" != 600 ]; then fail "proxy.env is mode $(stat -c %a "$rt/proxy.env"), expected 600"; fi +expect_no_file "$rt/proxy.env.new" +expect_no_file "$rt/secret.new" +expect_no_file "$rt/bao.err" +expect_secret_unprinted + +setup PROXY "no credential minted yet writes nothing" +kv_rc=2 +kv_err="No value found at secret/data/swarm/agents/a1" +go +expect_rc 0 +expect_err "no egress-proxy credential at $proxy_path yet" +expect_calls "$login_call" "$proxy_kv_call" +expect_no_file "$rt/proxy.env" +expect_no_file "$rt/proxy.env.new" +expect_no_file "$rt/secret.new" + +setup PROXY "no credential minted yet leaves the file in place" +proxy_env old.example >"$rt/proxy.env" +chmod 0600 "$rt/proxy.env" +kv_rc=2 +go +expect_rc 0 +expect_file "$rt/proxy.env" "$(proxy_env old.example)" + +setup PROXY "a store that refuses the login leaves the file in place" +proxy_env old.example >"$rt/proxy.env" +login_rc=2 +login_err=$'Error authenticating: Error making API request.\n\nCode: 503. Errors:\n\n* Vault is sealed\n' +go +expect_rc 1 +expect_err "could not log in to the swarm secret store" +expect_err "Vault is sealed" +expect_calls "$login_call" +expect_file "$rt/proxy.env" "$(proxy_env old.example)" + +setup PROXY "a store that does not answer leaves the file in place" +proxy_env old.example >"$rt/proxy.env" +login_rc=2 +login_err="Error authenticating: Put \"$bao_addr/v1/auth/cert/login\": dial tcp: lookup bao.t.local: no such host" +go +expect_rc 1 +expect_file "$rt/proxy.env" "$(proxy_env old.example)" + +setup PROXY "0600 files already in place do not block the fetch" +printf stale >"$rt/proxy.env.new" +printf stale >"$rt/secret.new" +printf stale >"$rt/bao.err" +go +expect_rc 0 +expect_no_err "Permission denied" +expect_file "$rt/proxy.env" "$(proxy_env "$proxy_no")" + +setup PROXY "an unchanged credential leaves the file alone" +proxy_env "$proxy_no" >"$rt/proxy.env" +before=$(stat -c %i "$rt/proxy.env") +go +expect_rc 0 +expect_out "are unchanged" +if [ "$(stat -c %i "$rt/proxy.env")" != "$before" ]; then fail "the unchanged file was replaced"; fi +expect_no_file "$rt/proxy.env.new" + +setup PROXY "a changed credential replaces the file whole" +proxy_env old.example >"$rt/proxy.env" +go +expect_rc 0 +expect_file "$rt/proxy.env" "$(proxy_env "$proxy_no")" + +setup PROXY "a value that is not a base64url secret is refused and the file kept" +proxy_env old.example >"$rt/proxy.env" +kv_out=$'abc\nNO_PROXY=*' +go +expect_rc 1 +expect_err "is not a base64url secret" +expect_file "$rt/proxy.env" "$(proxy_env old.example)" +if grep -qF abc "$dir/out" "$dir/err"; then fail "the refused value was printed"; fi + +setup PROXY "an empty value is refused" +kv_out= +go +expect_rc 1 +expect_err "is not a base64url secret" +expect_no_file "$rt/proxy.env" + +# A hive that predates the facts, or whose squid does not announce itself, gets +# no proxy: the store is not asked and nothing is written. +no_fact() { + expect_rc 0 + expect_err "this hive's facts name no squid proxy" + expect_calls + expect_no_file "$rt/proxy.env" + expect_secret_unprinted +} + +setup PROXY "a hive that writes no egress facts means no proxy" +printf 'HIVE_DASHBOARD_PORT=7000\nHYPERHIVE_HIVE_DOMAIN=t.hive\n' >"$dir/etc/hyperhive-hive-facts" +go +no_fact + +setup PROXY "no facts file means no proxy" +rm "$dir/etc/hyperhive-hive-facts" +go +no_fact + +setup PROXY "a proxy fact without the direct hosts means no proxy" +printf 'HIVE_SQUID_PROXY=10.42.0.1:3128\n' >"$dir/etc/hyperhive-hive-facts" +go +no_fact + +setup PROXY "direct hosts without a proxy fact mean no proxy" +printf 'HIVE_EGRESS_NO_PROXY=10.42.0.1,t.hive\n' >"$dir/etc/hyperhive-hive-facts" +go +no_fact + +for bad in 'not-an-address' 'http://10.42.0.1:3128' '10.42.0.1' 'proxy.t.local:3128'; do + setup PROXY "a proxy fact that is not an address and a port is refused: $bad" + printf 'HIVE_SQUID_PROXY=%s\nHIVE_EGRESS_NO_PROXY=10.42.0.1\n' "$bad" >"$dir/etc/hyperhive-hive-facts" + go + no_fact +done + +setup PROXY "direct hosts that are not host names are refused" +printf 'HIVE_SQUID_PROXY=10.42.0.1:3128\nHIVE_EGRESS_NO_PROXY=a b,c\n' >"$dir/etc/hyperhive-hive-facts" +go +no_fact + +setup PROXY "no facts leave the file in place" +proxy_env old.example >"$rt/proxy.env" +rm "$dir/etc/hyperhive-hive-facts" +go +expect_rc 0 +expect_file "$rt/proxy.env" "$(proxy_env old.example)" + +setup PROXY "the direct hosts are used as the hive wrote them, after the loopback names" +printf 'HIVE_SQUID_PROXY=10.9.8.7:3129\nHIVE_EGRESS_NO_PROXY=10.9.8.7,x.t.hive,y.t.hive\n' >"$dir/etc/hyperhive-hive-facts" +go +expect_rc 0 +expect_file "$rt/proxy.env" "$(proxy_url=http://a1:the-secret@10.9.8.7:3129 proxy_env localhost,127.0.0.1,::1,10.9.8.7,x.t.hive,y.t.hive)" + if [ "$failures" -ne 0 ]; then echo "script-test-agent-bao-fetch: $failures failed assertions over $count runs" >&2 exit 1 diff --git a/nix/script-tests/agent-hive-facts.sh b/nix/script-tests/agent-hive-facts.sh index 2d1b3517..636a8be7 100644 --- a/nix/script-tests/agent-hive-facts.sh +++ b/nix/script-tests/agent-hive-facts.sh @@ -40,7 +40,7 @@ check() { for sh in $SHELLS; do # Only what the snippet exported reaches the child's environment. No PATH: # the snippet must not need one. - got=$(env -i "$sh" -c ". '$dir/profile'; exec '$envbin'" | grep -E '^(HYPERHIVE_|HIVE_DASHBOARD_PORT=|HIVE_AGENT_NATS_URL=|CLAUDE_|LD_PRELOAD=|__hyperhive)' | sort) + got=$(env -i "$sh" -c ". '$dir/profile'; exec '$envbin'" | grep -E '^(HYPERHIVE_|HIVE_DASHBOARD_PORT=|HIVE_AGENT_NATS_URL=|HIVE_SQUID_PROXY=|HIVE_EGRESS_NO_PROXY=|CLAUDE_|LD_PRELOAD=|__hyperhive)' | sort) if [ "$got" != "$(printf '%s\n' "$want" | sort)" ]; then fail "$sh: a child of the sourcing shell saw:"$'\n'"$got" fi @@ -68,6 +68,13 @@ HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES=team=a,env=prod HIVE_AGENT_NATS_URL=tls://nats.example.test:4222 CLAUDE_REMOTE_CONTROL_SESSION_NAME_PREFIX=unknown-a1" +# The egress facts are for ./egress-proxy.nix's fetch, which reads the file +# itself: a unit with `HIVE_SQUID_PROXY` in its environment would hold a proxy +# address with no credential to go with it. +printf 'HYPERHIVE_HIVE_NAME=pr1ma\nHIVE_SQUID_PROXY=10.42.0.1:3128\nHIVE_EGRESS_NO_PROXY=10.42.0.1,pr1ma.example.test\n' >"$facts" +check "the egress facts stay out of the environment" "HYPERHIVE_HIVE_NAME=pr1ma +CLAUDE_REMOTE_CONTROL_SESSION_NAME_PREFIX=pr1ma-a1" + : >"$facts" check "an empty file (no facts)" "CLAUDE_REMOTE_CONTROL_SESSION_NAME_PREFIX=unknown-a1" diff --git a/nix/vm-tests/hive-squid.nix b/nix/vm-tests/hive-squid.nix index 735bf3d0..7824985f 100644 --- a/nix/vm-tests/hive-squid.nix +++ b/nix/vm-tests/hive-squid.nix @@ -1,5 +1,7 @@ # `checks.vm-hive-squid`: runs ../host-modules/hive-squid.nix in a VM, which -# the module-eval suite can only read. Three nodes on two networks: +# the module-eval suite can only read, and checks that an agent with the proxy +# variables of nix/agent-modules/egress-proxy.nix reaches squid and skips it +# where it should. Three nodes on two networks: # # agent (vlan 1) --> proxy (vlan 1 = the stand-in bridge, vlan 2 = uplink) --> web (vlan 2) # @@ -170,6 +172,8 @@ pkgs.testers.runNixOSTest { value = [ p.name ]; }) privateNets ); + # The gateway: nginx answers on the bridge, which agents reach directly. + networking.firewall.interfaces.eth1.allowedTCPPorts = [ 443 ]; # Renders the users file the way hive-c0re does. environment.systemPackages = [ pkgs.mkpasswd ]; services.hyperhive.squid.selfcheck.unrestrictedCanary = "other.t.local"; @@ -211,6 +215,8 @@ pkgs.testers.runNixOSTest { { lib, ... }: { virtualisation.vlans = [ 1 ]; + # The bridge listener, by the name the proxy resolves to the same address. + networking.hosts.${bridgeIp} = [ "bridge.t.local" ]; networking.interfaces.eth1.ipv4.addresses = lib.mkForce [ { address = "192.168.1.3"; @@ -357,6 +363,45 @@ pkgs.testers.runNixOSTest { "http://${bridgeIp}:3128/squid-internal-mgr/counters) = 403" ) + # The variables an agent's `proxy.env` carries (nix/agent-modules/egress-proxy.nix), + # as a curl on the agent reads them. + def agent_env(cred, no_proxy): + url = f"http://{cred}@${bridgeIp}:3128" + pairs = { + "HTTPS_PROXY": url, "HTTP_PROXY": url, "https_proxy": url, "http_proxy": url, + "NO_PROXY": no_proxy, "no_proxy": no_proxy, + } + return " ".join(f"{k}='{v}'" for k, v in pairs.items()) + + def agent_get(cred, no_proxy, url): + # The status of the answer, whoever gave it; 000 when none came. + return agent.succeed( + f"env {agent_env(cred, no_proxy)} curl -sS -m 15 -o /dev/null -w '%{{http_code}}' {url} 2>/dev/null || true" + ).strip() + + def squid_saw(pattern): + return re.search(pattern, proxy.succeed("journalctl -u squid.service -o cat")) + + with subtest("an agent with the proxy variables reaches an allowed name through squid and a swarm-internal name directly"): + NO_PROXY = "localhost,127.0.0.1,::1,bridge.t.local,${bridgeIp}" + # Plain HTTP to the listeners' port 443, so the answer carries a status. + assert agent_get(ATLAS, NO_PROXY, "http://forge.t.local:443/") not in ("000", "403", "407") + assert squid_saw(r"192\.168\.1\.3 TCP_\w+/\d+ .* GET http://forge\.t\.local:443/ atlas "), "through squid, as atlas" + assert agent_get(ATLAS, NO_PROXY, "http://bridge.t.local:443/") not in ("000", "403", "407") + assert not squid_saw(r"GET http://bridge\.t\.local:443/ "), "a swarm-internal name never reached squid" + # Control: without the exemption the same request goes to squid, which refuses the bridge. + assert agent_get(ATLAS, "localhost", "http://bridge.t.local:443/") == "403" + assert squid_saw(r"192\.168\.1\.3 TCP_DENIED/403 .* GET http://bridge\.t\.local:443/ "), "control: squid refuses it" + # CONNECT takes the same path: HTTPS_PROXY names the proxy for https URLs. + assert agent.succeed( + f"env {agent_env(ATLAS, NO_PROXY)} curl -sS -m 15 -o /dev/null -w '%{{http_connect}}' https://forge.t.local/ 2>/dev/null || true" + ).strip() == "200" + + with subtest("an agent without the proxy variables goes direct"): + assert agent.succeed( + "curl -sS -m 15 -o /dev/null -w '%{http_code}' http://bridge.t.local:443/ 2>/dev/null || true" + ).strip() not in ("000", "403", "407") + with subtest("the self-check passes with its credential"): proxy.succeed("install -d -m 0700 /run/credstore") proxy.succeed(f"printf '%s\\n' {PW['selfcheck.squid']} > /run/credstore/hive-squid.selfcheck")