diff --git a/CLAUDE.md b/CLAUDE.md index c38666f9..5b8917fc 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -202,6 +202,13 @@ hand-maintained per-file tree drifts out of sync with the code. for agents: nginx sends it `/select/logsql/` requests carrying an agent token, it answers 200 or 401 via `swarm-agent-verify`. Runs on the log store's host (`swarm-victorialogs.nix`). +- **`swarm-builder/`** — the swarm builder's build daemon, in its own + container where `deploy.builder.enable` is set. Takes an agent config PR's + head from `swarm-controller` over the queue, builds it on its host's agent + base through the host nix daemon (`nix/builder/check-flake`), and replies + with the store path or the error tail; the controller writes the commit + status. Holds a read-only forge token. Design: + [`docs/swarm/builder.md`](docs/swarm/builder.md). - **`swarmctl/`** — swarm-level operator CLI, installed by the swarm-controller module on the host that runs the daemon. Runs as **root and acts directly** — it serves no socket, publishes no HTTP diff --git a/Cargo.lock b/Cargo.lock index d8b08839..c9e8d083 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4829,6 +4829,21 @@ dependencies = [ "serde_json", ] +[[package]] +name = "swarm-builder" +version = "0.1.0" +dependencies = [ + "anyhow", + "async-nats", + "futures-util", + "hive-log", + "serde_json", + "swarm-queue-client", + "swarm-secret-client", + "tokio", + "tracing", +] + [[package]] name = "swarm-controller" version = "0.1.0" diff --git a/Cargo.toml b/Cargo.toml index 7d8112bd..00087010 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -27,6 +27,7 @@ members = [ "hivectl", "swarm-agent-verify", "swarm-authelia-bridge", + "swarm-builder", "swarm-authelia-bridge-sock", "swarm-controller", "swarm-matrix-client", diff --git a/docs/README.md b/docs/README.md index b026e8ee..0121a33e 100644 --- a/docs/README.md +++ b/docs/README.md @@ -89,6 +89,8 @@ declarations. - **How will a PR get checked against its repo's own yes/no rules before a full review?** → [`swarm/review-classifier.md`](swarm/review-classifier.md) (planned, not implemented yet). +- **Does an agent config PR build before it's merged?** → + [`swarm/builder.md`](swarm/builder.md). ## Scheduler, CI, observability diff --git a/docs/swarm/builder.md b/docs/swarm/builder.md new file mode 100644 index 00000000..1ad34452 --- /dev/null +++ b/docs/swarm/builder.md @@ -0,0 +1,81 @@ +# Swarm builder + +The swarm builder checks that an agent config PR builds before an operator +merges it. Each time an `agent-configs/` PR opens, reopens or gets a new +push, the builder builds the PR's head on the agent base, and the result shows as a +`swarm-builder` commit status on the PR and as a `build_agent` node on the +swarm UI's Jobs page. + +Where it runs and how it's enabled: [`services.md`](services.md#builder). + +## What happens on a config PR + +1. The forge delivers the `pull_request` event on the `config-pr` webhook + swarm-controller already registers on `agent-configs`. An `opened`, + `reopened` or `synchronized` action on an open PR queues one `build_agent` + node carrying the agent, the head commit and the PR number. An edit, a + label or a close builds nothing. +2. The node sets the status to `pending`, then sends the build as a request on + the queue subject `$SWARM.builder.build` and waits up to 61 minutes for the + reply. +3. The builder runs `nix build` of `nix/builder/check-flake`'s `check` output + with every input overridden: + - `hyperhive` and `nixpkgs`: the sources the builder's own host was + deployed with, the same two a hive on that host renders into its meta + flake (`c0re.hyperhiveFlake`, `c0re.nixpkgsFlake`). + - `agent`: `git+https:///agent-configs/?ref=refs/pull//head&rev=`. + A build has one hour. +4. The reply is the system closure's store path, or the last lines `nix` + printed. The node writes the status on the head commit: + + | Outcome | Status | Description | Node | + | ----------------------------------------- | --------- | --------------------------- | ------ | + | built | `success` | the store path | done | + | `nix` failed or timed out | `failure` | the last line `nix` printed | failed | + | no builder listening, or no reply in time | `error` | why | failed | + + The full tail is in the failed node's error on the Jobs page and in the + builder's journal line for the build. + +## What the check is, and isn't + +- It composes the config the way a hive's meta flake does + (`agent-base.extendModules` with the config's `nixosModules.default`), minus + what a hive adds: its CA bundle, docs source, `claude-code` path, ports and + OTEL settings, and the `follows` that point the config's own inputs at the + hive's. A green check can therefore still fail on a hive, and the reverse. +- The agent base is the builder's host's, not the hive's. Hives on another + hyperhive or nixpkgs pin build something else. +- The builder pushes nothing and records nothing. The closure stays in the builder + host's store until it's collected, so on a host that also runs the hive, the + hive's own build can reuse whatever matches. +- It doesn't gate anything. A merge deploys whatever the status says. +- A lost webhook delivery isn't retried: the head isn't built until the next + push. The jobs graph is in memory, so a controller restart drops a build in + flight and leaves its status `pending`. + +## Credentials + +| What | Where | Who reads it | +| ----------------------- | ------------------------------------------------- | ---------------------------------------- | +| store leaf | `deploy.bao.builderClient{Cert,Key}File` | nspawn, into the service as a credential | +| queue client secret | `secret/swarm/services/swarm-builder/oidc/client` | the builder, at start | +| forge token (read-only) | `secret/swarm/services/builder/forge/token` | the builder, on every build | + +The token belongs to the `builder` forge account, which swarm-controller +creates, puts in the `hives` team of `agent-configs` (read on every config +repo) and mints a `read:repository` token for, in a five-minute pass. `git` +gets the token through a credential helper reading the child's environment: +it's in no argument, URL or lock, so no line `nix` prints can carry it into a +status. The builder holds no write credential; swarm-controller writes the +status with its own forge token. Details per path: +[`credentials.md`](credentials.md). + +## Isolation + +The container has its own network namespace on the bridge, as the CI runner +does, so a config's evaluation can't reach a host-loopback service. Evaluation +is pure and runs in the container. Builds go to the host's nix daemon as an +untrusted client: a config can set no daemon option, and its derivations build +in the host's sandbox. A fixed-output derivation still has the network, as it +does for any build on that host. diff --git a/docs/swarm/credentials.md b/docs/swarm/credentials.md index c3101d53..4b028bfa 100644 --- a/docs/swarm/credentials.md +++ b/docs/swarm/credentials.md @@ -55,28 +55,29 @@ of the cell says how. -| store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull | -| ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `swarm/agents//matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again | -| `swarm/agents//matrix/` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | must be stated | -| `swarm/agents//matrix-recovery/` | `swarm-controller`, a random passphrase for every account listed under the agent's `matrix/`, in a five-minute pass | the agent container itself, under the certificate its hive passed in, when `hive-matrix-daemon` brings the account up | ❌ never replaced: the account's server-side key backup opens only with the passphrase it was enabled with, so the pass writes only to an empty path | ❌ read when the account comes up; an account whose passphrase lands later turns backup on at the next daemon start | -| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass | -| `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, at start and on refusal | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read at start and held in memory; read again when authelia refuses it, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value | -| `swarm/controller/swarm-controller/forge/token` | the forge, inside the `hive-forge` container, once; `swarm-bao-forge-publisher` on the forge's host copies it in under its own certificate, with a write-only grant | `swarm-controller`, under its own certificate, at start | ❌ the forge mints it once and keeps its copy in the container; the publisher writes that copy each time the forge container starts and each time the mint writes the file | ❌ the controller reads it at start, and in the background with backoff until it first finds it; a replaced token reaches it with the next restart | -| `swarm/agents//bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config; then `hive-agent-bao-identity` in the agent container, under the leaf it holds | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ✅ `hive-agent-bao-identity` re-reads it every 10 minutes and swaps a new leaf into `/run/hive-agent-bao-identity/`, which every store login in the container presents. It reads under the leaf it holds, so it renews only while that leaf is valid | -| `swarm/agents//queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive; `swarm-logs` in the agent container, under the same certificate, on every query, as the bearer it presents at the log store; `swarm-nats-auth` and `swarm-logs-auth`, each under its own certificate, to check a presented secret | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff. `swarm-logs` reads it on every query and once more after a 401. `swarm-logs-auth` keeps granting a secret it verified for `deploy.victorialogs.authVerdictTtl` (60 s by default) without reading the path, so a replaced or deleted secret still works at the log store for up to that long | -| `swarm/agents//forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer | -| `swarm/agents//forge/