send agent traffic through squid once its hive announces it
This commit is contained in:
atlas
parent
86e9f49fe6
commit
b551eb02ff
5 changed files with 885 additions and 27 deletions
|
|
@ -45,7 +45,7 @@ that touches it.
|
|||
| container | netns | IPv4 | listens / reached via |
|
||||
| ------------------------- | ----------------------- | -------------- | ----------------------------------------------------------------------------------------------- |
|
||||
| gateway (host) | host | host addresses | nginx `:80`/`:443` (every vhost); dnsmasq `bridgeIp:53` + DHCP `:67` on the bridge |
|
||||
| egress proxy (host) | host | `bridgeIp` | squid `bridgeIp:3128`; no agent uses it yet |
|
||||
| egress proxy (host) | host | `bridgeIp` | squid `bridgeIp:3128`; agents with a credential send HTTP and HTTPS through it |
|
||||
| `hive-forge` | host (shared) | host addresses | forgejo `:3000` http, `:2222` git-ssh; fronted by the `forge.<swarm-domain>` vhost |
|
||||
| `hive-matrix` | private, veth on bridge | DHCP pool | tuwunel on UDS `/run/hive-matrix/tuwunel.sock` → the matrix vhost; out to store, SSO, peers |
|
||||
| `swarm-ci-runner` | private, veth on bridge | DHCP pool | outbound only (runner → forge); no inbound surface |
|
||||
|
|
@ -212,9 +212,11 @@ DNS/gateway), so only expose services safe for any agent to reach.
|
|||
|
||||
Every hive host runs squid in the host netns, listening on `<bridgeIp>:3128`
|
||||
(`nix/host-modules/hive-squid.nix`). The port is open on the bridge interface
|
||||
only. **No agent uses it yet**: none sends a credential, and agent traffic
|
||||
still goes straight out through the NAT. The proxy goes first so you can check
|
||||
its metrics and logs before anything reroutes.
|
||||
only. An agent sends its outbound HTTP and HTTPS through it once its hive says
|
||||
the proxy is ready and it has fetched its proxy credential (see _Agents and the
|
||||
proxy_ below). Any other agent goes straight out through the NAT, as before,
|
||||
and so does any program that ignores the proxy variables: nothing blocks direct
|
||||
egress yet.
|
||||
|
||||
Squid filters on the destination hostname: a `CONNECT` target or a plain-HTTP
|
||||
host. It never decrypts TLS, so it holds no signing key. It caches nothing.
|
||||
|
|
@ -233,6 +235,42 @@ host. It never decrypts TLS, so it holds no signing key. It caches nothing.
|
|||
A request without a valid credential gets a 407, once the destination
|
||||
checks below have passed it.
|
||||
|
||||
- **Agents and the proxy.** An agent uses the proxy only when its hive says
|
||||
this host's squid is ready for it. `hive-c0re` puts two facts in every
|
||||
container's `/etc/hyperhive-hive-facts`, only on a hive that holds a store
|
||||
identity: `HIVE_SQUID_PROXY` (`services.hyperhive.squid.listen`, the bridge IP
|
||||
and port) and `HIVE_EGRESS_NO_PROXY` (the direct hosts below). A hive that
|
||||
predates them writes neither, and its agents go direct.
|
||||
`hive-agent-proxy-credential.service` (`nix/agent-modules/egress-proxy.nix`)
|
||||
runs in each agent's container, as the agent user, at start and every 10
|
||||
minutes. With both facts present it logs in to the secret store with the
|
||||
agent's own certificate, reads `swarm/agents/<agent>/proxy-credential` and
|
||||
writes `/run/hive-agent-proxy-credential/proxy.env` (`0600`) with
|
||||
`HTTPS_PROXY`, `HTTP_PROXY`, their lower-case forms, `NODE_USE_ENV_PROXY=1`
|
||||
and `NO_PROXY`/`no_proxy`. The proxy URL is
|
||||
`http://<agent>:<credential>@<bridgeIp>:3128`; Claude Code, Node, curl, git
|
||||
and reqwest send the userinfo as `Proxy-Authorization`. The harness, its bash
|
||||
tool, the subagents, the MCP servers and the pollers load the file with
|
||||
`EnvironmentFile=-` when they start. Until the facts and the credential both
|
||||
exist nothing writes the file, so those units have no proxy variable and go
|
||||
direct; a failed read keeps the file already in place. Nothing restarts on
|
||||
deploy: an agent takes the proxy at its next start, which is its next rebuild
|
||||
or restart. The credential is in the environment of every process those units
|
||||
start, so `env` in the agent's shell prints it. Login shells
|
||||
(`nixos-container root-login`, SSH) don't load the file.
|
||||
- **Direct destinations.** `HIVE_EGRESS_NO_PROXY` is the hosts this agent must
|
||||
reach without the proxy, and `NO_PROXY` is `localhost,127.0.0.1,::1` followed
|
||||
by that list. `hive-c0re` computes it from the host's options: the bridge IP,
|
||||
the hive's domain (`services.hyperhive.domain`), the secret store and queue
|
||||
names (`services.hyperhive.swarm.bao.domain`, `swarm.nats.domain`),
|
||||
`services.hyperhive.swarm.serviceDomains` (the swarm UI and controller apex,
|
||||
the forge, matrix, attic, logs, metrics and otel names) and every name the
|
||||
gateway answers with the bridge IP (`services.hyperhive.gateway.localNames`).
|
||||
Squid refuses every destination that resolves to the bridge, and the hive's
|
||||
resolver answers the bridge address for those names, so they never go through
|
||||
the proxy. A bare name matches itself and its subdomains. Node's `NO_PROXY`
|
||||
has no CIDR support, so the list holds the bridge as an address.
|
||||
|
||||
- **Two lists.** The credential decides the policy. `hive-c0re` writes a
|
||||
second file beside the users file, `services.hyperhive.squid.unrestrictedFile`
|
||||
(`/var/lib/hive-squid/unrestricted`, `hive-core:squid`, `0640`): the names of
|
||||
|
|
|
|||
|
|
@ -43,10 +43,10 @@ agent's build, and neither do the hive's other per-host values: the dashboard
|
|||
port (`HIVE_DASHBOARD_PORT`), the pinned claude-code
|
||||
(`HYPERHIVE_CLAUDE_CODE_PATH`), extra OTEL labels
|
||||
(`HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES`) and the queue's address
|
||||
(`HIVE_AGENT_NATS_URL`, from `deploy.hive-controller.queue.agentNatsUrl`). `hive-priv` writes them to
|
||||
(`HIVE_AGENT_NATS_URL`, from `deploy.hive-controller.queue.agentNatsUrl`) and the egress proxy's address and direct hosts (`HIVE_SQUID_PROXY`, `HIVE_EGRESS_NO_PROXY`, see [the egress proxy](../networking/network.md#egress-proxy-squid)). `hive-priv` writes them to
|
||||
`/etc/hyperhive-hive-facts` before every container start, leaving out any the
|
||||
hive doesn't set; the agent then uses its own default. The agent's environment
|
||||
generator (`nix/agent-modules/hive-facts.nix`) passes them to every unit,
|
||||
generator (`nix/agent-modules/hive-facts.nix`) passes all but the two egress facts, which only the proxy fetch reads, to every unit,
|
||||
including `hivectl choom` sessions, and `/etc/set-environment` exports them in login
|
||||
shells that no unit starts (`nixos-container root-login`). None is part of
|
||||
an agent's closure, and an agent started on another hive reports that hive.
|
||||
|
|
@ -323,7 +323,7 @@ that value. Read it with
|
|||
`swarm_queue_client::agent_permissions::unrestricted`. The hive reads it for
|
||||
the agent's squid policy: [the egress proxy](../networking/network.md#egress-proxy-squid)
|
||||
lets an unrestricted agent's credential reach any destination, and holds the
|
||||
others to the allow-list. No agent sends a credential yet.
|
||||
others to the allow-list. An agent sends its credential once it has fetched it, from its next start.
|
||||
|
||||
**How a change applies** —
|
||||
- The agent's harness reads its own key every 10 seconds and builds every
|
||||
|
|
|
|||
|
|
@ -36,7 +36,7 @@ the rest of the cell says how.
|
|||
| `swarm/agents/<agent>/claude-credentials` | `swarm-claude-login`, when you log the agent in from its card; one version per login | the agent, while it waits for a login | ❌ log in again | ✅ every 30 seconds while waiting |
|
||||
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, every 5 minutes | `swarm-controller` and the hive | ✅ when missing or rejected by the homeserver | ✅ the hive re-reads on every sweep |
|
||||
| `swarm/hives/<hive>/forge/token` | `swarm-controller`, every 5 minutes | `swarm-controller` and the hive | ✅ when missing or no longer matching the forge | ✅ read on every forge call |
|
||||
| `swarm/agents/<agent>/proxy-credential` | `swarm-controller`: 32 random bytes, at start and every 5 minutes for every agent that holds a store identity | the hive (`hive-c0re`); the agent can read it, nothing sends it yet | ❌ never replaced: written only while the path holds nothing | ✅ `hive-c0re` re-reads it every 30 seconds |
|
||||
| `swarm/agents/<agent>/proxy-credential` | `swarm-controller`: 32 random bytes, at start and every 5 minutes for every agent that holds a store identity | the hive (`hive-c0re`) and the agent, which fetches it | ❌ never replaced: written only while the path holds nothing | ✅ `hive-c0re` re-reads it every 30 seconds |
|
||||
| `swarm/hives/<hive>/squid-selfcheck` | `swarm-controller`, in the same pass as the agents' credentials | the hive: `hive-c0re` and `squid-selfcheck-credential` (root) | ❌ never replaced: written only while the path holds nothing | ✅ `hive-c0re` every 30 seconds, the root unit every 10 minutes |
|
||||
| `swarm/hives/<hive>/oidc/client` | authelia, first boot; `swarm-secret-publish` copies it in | the hive | ❌ minted once; `swarm-secret-publish` republishes a re-mint | ✅ re-read when authelia refuses it |
|
||||
| `swarm/services/<clientId>/oidc/client` | authelia | that service | must be stated | must be stated |
|
||||
|
|
@ -130,6 +130,7 @@ pieces fit, and the reasoning behind them, is in [`credentials.md`](credentials.
|
|||
| the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | the swarm secret store at `swarm/controller/swarm-controller/forge/token`, written by the mint under the forge publisher's leaf, which the forge container loads as a credential; the controller holds it in memory only. `/var/lib/forgejo/swarm-controller-token-name` in the forge container holds its name, the minter's record that it has minted | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) |
|
||||
| the swarm controller's Grafana token | `swarm-bao-grafana-controller-token.service`, **on the host that runs Grafana**, as Grafana's built-in admin over its socket: it creates the `swarm-controller` service account as a Viewer (and sets it back to Viewer on every run), mints a token for it and writes it to the swarm secret store at `swarm/controller/swarm-controller/grafana/token` under Grafana's leaf. Re-run every ten minutes: it keeps a stored token Grafana accepts, and replaces one Grafana refuses with 401 (or a missing one), deleting every other token of that account first | the swarm secret store only; the controller reads it for `GET /api/alerts` and holds it in memory, re-reading the store once when Grafana answers 401 | nothing to place: Grafana's leaf is already required to run Grafana. Until the unit has run, `/api/alerts` answers 503 naming the empty path |
|
||||
| an agent's API-key environment (`ANTHROPIC_API_KEY`, `ANTHROPIC_BASE_URL`, an ACP provider key) | `hive-agent-backend-env.service`, in the agent's container, on an agent with a store that uses `useApiKey` or the `acp` runtime. When the store holds nothing at `swarm/agents/<agent>/backend-env` and `backendEnvironmentFile` is a non-empty file, it creates the leaf with the file's exact bytes and `-cas=0`, under the agent's own leaf, which may create that path and not update it. It never rotates a value and never rewrites or deletes the file. To set or replace a key, use **set the API-key environment** in the swarm UI (`PUT /api/hives/{hive}/agents/{agent}/backend-env`): `swarm-controller` overwrites the leaf with its own grant, and the store's value then wins over the file. Its **remove** (`DELETE` on the same path) deletes the leaf, and an agent whose file has content then creates it from that file again | the swarm secret store at that path; the unit writes it to `/run/hive-agent-backend-env/env` (tmpfs, `0600`, root), and `hive-agent` and `hive-subagent-daemon` load that after `backendEnvironmentFile` as `EnvironmentFile=` | the same unit, same path. A variable in the store wins over the same one in the file. With the store down at a start, the file alone carries the agent, as it did before the store held a copy; with an older hive build the agent ignores the leaf and keeps reading the file |
|
||||
| an agent's proxy settings (`HTTPS_PROXY`, `HTTP_PROXY`, `NO_PROXY`, the credential in the proxy URL) | `hive-agent-proxy-credential.service`, in the agent's container, as the agent user, at start and every 10 minutes: it reads `swarm/agents/<agent>/proxy-credential` under the agent's own leaf and builds the file from it. It writes nothing while the path holds nothing or the hive's facts name no squid | `/run/hive-agent-proxy-credential/proxy.env` (tmpfs, `0600`, the agent user); the harness, its bash tool, the subagents, the MCP servers and the pollers load it as `EnvironmentFile=-` and read it when they start | With no credential, or on a hive that writes no `HIVE_SQUID_PROXY` and `HIVE_EGRESS_NO_PROXY` fact, nothing writes the file and a unit has no proxy variable and reaches the internet directly. A failed read keeps the file already in place. A unit started before the file appeared goes direct until its next start. The credential is also in the environment of every process those units start, so `env` in the agent's shell prints it |
|
||||
| an agent's claude login (`.credentials.json`) | `swarm-claude-login.service`, in its container, when an operator completes `claude auth login` from the agent's card in the swarm UI; it stores the file as a new version at `swarm/agents/<agent>/claude-credentials` under its own leaf, which may create and update that path and read nothing under an agent | the swarm secret store at that path; the agent's `hive-agent` reads it under the agent's own leaf every 30 seconds while it waits for a login and writes it to `~/.claude/.credentials.json` | the same read, co-located or not; nothing for an operator to place beyond the agent's store leaf. When the store's copy and the agent's differ the KV version decides: the agent installs only a version newer than the last it installed (`claude-credentials-version` under the harness dir), because claude rotates the refresh token in its own copy and the stored one goes stale. The agent ignores a stale or equal version, and a relogin writes a newer one — [claude-login.md](claude-login.md) |
|
||||
| the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
|
||||
| the queue container's journal forwarder's copy of the queue's OIDC secret | `swarm-journal-forwarder-oidc.service` inside the `swarm-nats` container reads it out of the swarm secret store under the responder's store leaf, and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. After a reboot the collector restarts until the file exists, then resumes from its saved journal cursor. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
|
||||
|
|
@ -156,7 +157,7 @@ pieces fit, and the reasoning behind them, is in [`credentials.md`](credentials.
|
|||
| the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/run/hive-forge-oidc/<id>.secret` on the host (tmpfs), bound into the forge container, handed to `forgejo-sso-source` through `LoadCredential` |
|
||||
| the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/run/hive-matrix-oidc/<id>.secret` on the host (tmpfs), bound into the homeserver container, handed to tuwunel through `LoadCredential` |
|
||||
| this hive's forge read token | `swarm-controller`, into the secret store at `swarm/hives/<hive>/forge/token`, for the `hive-<hive>` account; a pass at start and every five minutes re-mints a missing or stale token | `hive-c0re` reads it under the hive's own store identity, and copies it (account on the first line, token on the second) to `/run/hyperhive/forge-hive-token` (tmpfs, `0600` `hive-core`) for nix's `git`, which `hive-priv` also runs. Each `internal/meta` read and each boot rewrites the copy when it differs; where they differ after a rotation, a build that starts before the next read fails to authenticate, and a retry reads the new token |
|
||||
| squid's users file | `hive-c0re` (`workers::squid_users`), every 30 seconds, from the proxy credentials of its agents and the self-check | `services.hyperhive.squid.usersFile` (`/var/lib/hive-squid/users`, `hive-core:squid`, `0640`): `<user>:<hash>` lines, SHA-512 crypt hashes only, renamed into place when it changed. `unrestricted` beside it lists the users whose agent holds the swarm's `unrestricted_internet` flag, names only; it lists every agent the hive never read as off `systemd-tmpfiles` creates it empty so squid's helper never sees a missing file. A hive whose agents have no credential yet (an older controller) has an empty file: every login fails and the self-check reports `ok=false`; no agent sends a credential, so no traffic changes |
|
||||
| squid's users file | `hive-c0re` (`workers::squid_users`), every 30 seconds, from the proxy credentials of its agents and the self-check | `services.hyperhive.squid.usersFile` (`/var/lib/hive-squid/users`, `hive-core:squid`, `0640`): `<user>:<hash>` lines, SHA-512 crypt hashes only, renamed into place when it changed. `unrestricted` beside it lists the users whose agent holds the swarm's `unrestricted_internet` flag, names only; it lists every agent the hive never read as off `systemd-tmpfiles` creates it empty so squid's helper never sees a missing file. A hive whose agents have no credential yet (an older controller) has an empty file: every login fails and the self-check reports `ok=false`; an agent without one sends none, so it goes direct |
|
||||
| the squid self-check's password | `swarm-controller` mints `swarm/hives/<hive>/squid-selfcheck`; `squid-selfcheck-credential` (root, every 10 minutes) reads it as the hive | `/run/credstore/hive-squid.selfcheck` (tmpfs, root, `0400`), imported by `squid-selfcheck.service` as a systemd credential and handed to `curl` on its stdin. Both self-check users (`selfcheck.squid` and the always-unrestricted `selfcheck-unrestricted.squid`) use it, so whoever holds it has unrestricted egress through this squid Missing, or different from the users file: the self-check logs `credential=absent` or a 407 and fails, and nothing else breaks |
|
||||
|
||||
### Store identities, one per principal
|
||||
|
|
|
|||
|
|
@ -569,8 +569,9 @@ the swarm never set the flag, which counts as on. The **restrict** or
|
|||
permission list along with the flag, because a `PUT` without the list resets a
|
||||
stored list to the default. An agent with no stored list sends the flag alone.
|
||||
The hive's egress proxy applies the flag to each agent's proxy credential (see
|
||||
[the egress proxy](../networking/network.md#egress-proxy-squid)); no agent sends
|
||||
one yet, so the flag changes no traffic today.
|
||||
[the egress proxy](../networking/network.md#egress-proxy-squid)). An agent sends its
|
||||
credential from its next start after it has fetched it, so a flag change applies
|
||||
to an agent's traffic from then on.
|
||||
|
||||
A saved change isn't instant. The hive reads the caps about every 30 seconds,
|
||||
and a running agent takes a changed cap at its next spawn or rebuild, so the
|
||||
|
|
|
|||
|
|
@ -209,6 +209,11 @@ token policy bounds file reads; network isolation bounds network reach.
|
|||
nixbld users can't read it. The swarm-minted token lives under
|
||||
`/run/hive-agent-forge-token/` (`0700` directory, `0400` file), outside
|
||||
the state dir.
|
||||
- `/run/hive-agent-proxy-credential/proxy.env` — the agent's egress-proxy
|
||||
credential inside its proxy URL (`0700` directory, `0600` file, tmpfs). Units
|
||||
load it as an environment file, so the credential is also in the environment of
|
||||
every process they start. It authenticates this agent to the hive's squid and
|
||||
nothing else.
|
||||
|
||||
**Policy**: all credential files written to agent state directories MUST be mode
|
||||
`0600` or stricter. Don't create world-readable secret files in agent state dirs.
|
||||
|
|
|
|||
Loading…
Reference in a new issue