diff --git a/docs/swarm/builder.md b/docs/swarm/builder.md index 34b384ec..f06bdb00 100644 --- a/docs/swarm/builder.md +++ b/docs/swarm/builder.md @@ -61,8 +61,9 @@ Where it runs and how it's enabled: [`services.md`](services.md#builder). ## Build records -`swarm-controller` records each build when it writes the final status: -built, failed, or no build ran or came back. The newest 100 are one JSON +`swarm-controller` records each config PR build when it writes the final status: +built, failed, or no build ran or came back. It also records each [deploy-gate +build](#deploy-gate) the builder answers, in the same list. The newest 100 are one JSON array under the key `recent` in the `builder-runs` KV bucket on the swarm queue, which only the controller reads or writes. It creates the bucket on first use and adds each build with a compare-and-set write, so the records @@ -72,8 +73,12 @@ toward the 100. Each record has: -- `id`: `--` -- `repo` (`agent-configs/`), `pr` and `head_sha` +- `id`: `--` for a + config PR's build, `meta---` + for a deploy's +- `repo` (`agent-configs/`), `pr` and `head_sha`; a deploy's build has + `repo` `internal/meta`, no `pr`, the `internal/meta` commit as `head_sha`, + and `agent`, the agent it built - `started_unix` and `finished_unix` - `outcome`: `built`, `failed` or `error`, the status's `success`, `failure` or `error` @@ -86,6 +91,13 @@ newest 100. Both are behind the swarm UI vhost's `admins` gate, and answer 503 when the controller has no swarm queue or can't open the bucket, 500 when it can't read it. +The controller records a deploy-gate build when the builder answers it, or when +asking it fails (no builder listening, no reply). A build that never got as far +as a request (no forge access, no `internal/meta` head) leaves no record, so a +controller that can't reach the forge doesn't fill the list with retries. Every +ten-minute retry of a failing agent is a record, and a version bump records one +build per agent, so the 100 cover fewer days when builds fail. + The final status links to `https:///jobs?run=`, the swarm UI's jobs tab, where the builds are a card under the job graph. Selecting **Details** on a red `swarm-builder` check opens that build's card, with `nix`'s tail, in the browser. The raw record diff --git a/docs/swarm/ui.md b/docs/swarm/ui.md index c198a189..c57085d6 100644 --- a/docs/swarm/ui.md +++ b/docs/swarm/ui.md @@ -27,7 +27,7 @@ shut. A dropdown inside a dialog opens above the dialog and the dialog doesn't c | `/agents//terminal` | one agent's live terminal | | `/agents//subagents//terminal` | one of that agent's subagents' live terminal | | `/reports` | two folded cards, each read only once opened: the **alerts** the swarm's Grafana holds now, the firing ones, with a button for the pending, no-data and error ones and a link to Grafana's own alert list; read through the controller with a Viewer service-account token, and a cross-repo **issue report** | -| `/jobs` | the controller's job graph, where agent creation and credential mints show progress, and under it the swarm builder's recent builds of agent config PRs, with what nix printed when one failed, and the review classifier's recent runs, a try-out on a real PR, and its settings | +| `/jobs` | the controller's job graph, where agent creation and credential mints show progress, and under it the swarm builder's recent builds (agent config PRs, and agents built from `internal/meta` before a deploy), with what nix printed when one failed, and the review classifier's recent runs, a try-out on a real PR, and its settings | | `/messages` | the swarm-wide message feed: every message between agents, and to them from the operator, reminders and compaction, with a history window and a filter on sender or recipient; a folded **broadcast** card above the feed sends one message to more than one agent at once, and a folded **schedules** card holds the scheduled prompts: add, edit, pause, resume and remove | | `/notifications` | the notification router's flows: a switch for each built-in, every flow drawn as a graph, and an editor for custom flows | @@ -498,9 +498,11 @@ shut, and each reads nothing until you open it; a link to a build opens the builds card. The old `/builds` address redirects there and keeps its query. -The **builds** card lists the agent config PR heads the swarm builder built, -newest first, from `GET /api/builder/runs`. Each card shows the repo and PR, -the first seven characters of the commit, how the build ended (**built**, +The **builds** card lists what the swarm builder built, newest first, from +`GET /api/builder/runs`: agent config PR heads, and the agents it builds from +`internal/meta` before each deploy (a version bump, the pin fan-out or a merged +config PR). Each card shows the repo and PR, or for a deploy's build the agent +and `internal/meta`, the first seven characters of the commit, how the build ended (**built**, **failed** or **error**), when it started and how long it took. Select a card to read it: for a failed build, what `nix` printed; for a built one, the system closure; for an error, why no build came back. diff --git a/frontend/packages/swarm-ui/src/pages/builds/BuildsPage.tsx b/frontend/packages/swarm-ui/src/pages/builds/BuildsPage.tsx index 50a138c0..909325f8 100644 --- a/frontend/packages/swarm-ui/src/pages/builds/BuildsPage.tsx +++ b/frontend/packages/swarm-ui/src/pages/builds/BuildsPage.tsx @@ -1,6 +1,6 @@ -// — the swarm builder's recent builds, newest first: which agent -// config PR heads it built, how each ended, and what `nix` printed when one -// failed. One card per build; selecting a card opens it. Reads +// — the swarm builder's recent builds, newest first: the agent +// config PR heads it built, and the agents it built from `internal/meta` before +// a deploy, how each ended, and what `nix` printed when one failed. One card per build; selecting a card opens it. Reads // `GET /api/builder/runs`. A `swarm-builder` commit status links to the jobs // page as `/jobs?run=`, which opens that build's card and scrolls to it. // The card sits under the job graph on that page; the old `/builds` address @@ -22,6 +22,7 @@ import { OUTCOME_LABEL, OUTCOME_TONE, bodyText, + runTitle, wantedRun, type BuildRun, } from "./builds-types.js"; @@ -56,9 +57,7 @@ function BuildCard({ onClick={onToggle} > - - {run.repo} #{run.pr} - + {runTitle(run)} {run.head_sha.slice(0, 7)}

- The agent config PR heads the swarm builder built, newest first: how - each ended, and what nix printed when one failed. Select a build to read - it. + What the swarm builder built, newest first: agent config PR heads, and + the agents it builds from internal/meta before a deploy. How each ended, + and what nix printed when one failed. Select a build to read it.

{error ? ( diff --git a/frontend/packages/swarm-ui/src/pages/builds/builds-types.ts b/frontend/packages/swarm-ui/src/pages/builds/builds-types.ts index 322ba6a5..b61f7b9e 100644 --- a/frontend/packages/swarm-ui/src/pages/builds/builds-types.ts +++ b/frontend/packages/swarm-ui/src/pages/builds/builds-types.ts @@ -10,9 +10,12 @@ export type BuildOutcome = "built" | "failed" | "error"; export interface BuildRun { /** `--`. */ id: string; - /** `agent-configs/`. */ + /** `agent-configs/` for a config PR's build, `internal/meta` for a deploy's. */ repo: string; - pr: number; + /** The config PR a build checks; a deploy's build has none. */ + pr?: number | null; + /** The agent a deploy's build built; a PR build's is in `repo`. */ + agent?: string | null; head_sha: string; started_unix: number; finished_unix: number; @@ -37,6 +40,16 @@ export const OUTCOME_TONE: Record = { error: "warning", }; +/** + * A run's card title: `agent-configs/atlas #3` for a config PR's build, + * `atlas from internal/meta` for a deploy's (the swarm builds every agent from + * `internal/meta` before sending its deploy). + */ +export function runTitle(run: BuildRun): string { + if (run.pr !== undefined && run.pr !== null) return `${run.repo} #${run.pr}`; + return run.agent ? `${run.agent} from ${run.repo}` : run.repo; +} + /** The run a `?run=` query names, or `null` when it names none. */ export function wantedRun(search: string): string | null { const id = new URLSearchParams(search).get("run"); diff --git a/frontend/packages/swarm-ui/test/builds-types.test.mjs b/frontend/packages/swarm-ui/test/builds-types.test.mjs index 2cba6d47..bbf71b68 100644 --- a/frontend/packages/swarm-ui/test/builds-types.test.mjs +++ b/frontend/packages/swarm-ui/test/builds-types.test.mjs @@ -3,7 +3,11 @@ // stripping, so the `.ts` module is imported as it is. import assert from "node:assert/strict"; import { test } from "node:test"; -import { bodyText, wantedRun } from "../src/pages/builds/builds-types.ts"; +import { + bodyText, + runTitle, + wantedRun, +} from "../src/pages/builds/builds-types.ts"; const base = { id: "atlas-abc-1", @@ -49,3 +53,20 @@ test("a run with no text for its outcome shows no body", () => { assert.equal(bodyText({ ...base, outcome: "failed" }), null); assert.equal(bodyText({ ...base, outcome: "built", error_tail: "x" }), null); }); + +test("a config PR's build is titled by its repo and PR, a deploy's by its agent", () => { + assert.equal( + runTitle({ ...base, outcome: "built" }), + "agent-configs/atlas #3", + ); + const deploy = { + ...base, + repo: "internal/meta", + pr: undefined, + agent: "atlas", + outcome: "built", + }; + assert.equal(runTitle(deploy), "atlas from internal/meta"); + assert.equal(runTitle({ ...deploy, pr: null }), "atlas from internal/meta"); + assert.equal(runTitle({ ...deploy, agent: null }), "internal/meta"); +}); diff --git a/swarm-controller/src/builder.rs b/swarm-controller/src/builder.rs index 45480538..1a101604 100644 --- a/swarm-controller/src/builder.rs +++ b/swarm-controller/src/builder.rs @@ -193,7 +193,8 @@ async fn record( job.rev.get(..12).unwrap_or(&job.rev) ), repo: format!("{CONFIG_ORG}/{}", job.agent), - pr: job.pr, + pr: Some(job.pr), + agent: None, head_sha: job.rev.clone(), started_unix, finished_unix: unix_now(), @@ -343,7 +344,7 @@ mod tests { panic!("one build recorded: {builds:?}") }; assert_eq!(build.repo, "agent-configs/atlas"); - assert_eq!((build.pr, build.head_sha.as_str()), (3, "abc")); + assert_eq!((build.pr, build.head_sha.as_str()), (Some(3), "abc")); assert_eq!(build.outcome, BuildOutcome::Failed); assert_eq!(build.error_tail.as_deref(), Some(tail)); assert_eq!((&build.out_path, &build.detail), (&None, &None)); diff --git a/swarm-controller/src/builder/runs.rs b/swarm-controller/src/builder/runs.rs index 8dcb655c..bf6377a4 100644 --- a/swarm-controller/src/builder/runs.rs +++ b/swarm-controller/src/builder/runs.rs @@ -51,10 +51,17 @@ pub(crate) struct BuildRecord { /// `--`, the last segment /// of the link on the build's status. pub(crate) id: String, - /// `agent-configs/`. + /// `agent-configs/` for a config PR's build, `internal/meta` for a + /// deploy's (`meta_build`). pub(crate) repo: String, - pub(crate) pr: u64, - /// The commit built. + /// The config PR a build checks. A `meta_build` build has none. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub(crate) pr: Option, + /// The agent a `meta_build` build built; a PR build's is in `repo`. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub(crate) agent: Option, + /// The commit built: the PR's head, or `internal/meta`'s for a deploy's + /// build. pub(crate) head_sha: String, /// Unix seconds: when the node sent the build, and when its answer came. pub(crate) started_unix: i64, @@ -362,7 +369,8 @@ pub(crate) mod tests { BuildRecord { id: id.to_owned(), repo: "agent-configs/atlas".to_owned(), - pr: 3, + pr: Some(3), + agent: None, head_sha: "abc".to_owned(), started_unix: 1_000, finished_unix: 1_060, diff --git a/swarm-controller/src/jobq.rs b/swarm-controller/src/jobq.rs index b46cb453..0d6c346d 100644 --- a/swarm-controller/src/jobq.rs +++ b/swarm-controller/src/jobq.rs @@ -655,7 +655,8 @@ async fn build_meta( agent: String, pr: Option, ) -> hive_jobq::scheduler::Outcome { - meta_build::run(forge.as_deref(), deps.queue.as_ref(), agent, pr).await + let builds = builder::runs::Builds::Live(deps.queue.clone()); + meta_build::run(forge.as_deref(), deps.queue.as_ref(), &builds, agent, pr).await } /// What an `UpdateMeta` pass decided: the deploys to send, and the placements diff --git a/swarm-controller/src/meta_build.rs b/swarm-controller/src/meta_build.rs index aeee6c7c..a13e64ec 100644 --- a/swarm-controller/src/meta_build.rs +++ b/swarm-controller/src/meta_build.rs @@ -35,6 +35,7 @@ use swarm_wire::agent_closures::AgentClosure; use swarm_wire::builder::{BuildFromMeta, FromMetaReply, META_SUBJECT, REPLY_DEADLINE}; use crate::agent_renewal::unix_now; +use crate::builder::runs::{BuildOutcome, BuildRecord, Builds}; use crate::forge::{self, objects::SHARED_ORG}; use crate::meta_flake::{META_REPO, MetaForge}; @@ -263,10 +264,66 @@ async fn clear_failure(head: &impl MetaRepo, pr: u64, agent: &str, meta_rev: &st } } -/// Build `agent` from `internal/meta`'s head and record the closure. +/// Store the build of `agent` from `meta_rev` that ended as `outcome`, `text` +/// being its store path, `nix`'s tail or the reason, in the list the builds +/// page shows beside the config PRs' builds. A failed write is a warning in +/// `Builds::record` and changes nothing about the node. +async fn record_run( + builds: &Builds, + agent: &str, + meta_rev: &str, + started_unix: i64, + outcome: BuildOutcome, + text: String, +) { + let mut build = BuildRecord { + id: format!( + "meta-{agent}-{}-{started_unix}", + meta_rev.get(..12).unwrap_or(meta_rev) + ), + repo: format!("{SHARED_ORG}/{META_REPO}"), + pr: None, + agent: Some(agent.to_owned()), + head_sha: meta_rev.to_owned(), + started_unix, + finished_unix: unix_now(), + outcome, + out_path: None, + error_tail: None, + detail: None, + }; + *match outcome { + BuildOutcome::Built => &mut build.out_path, + BuildOutcome::Failed => &mut build.error_tail, + BuildOutcome::Error => &mut build.detail, + } = Some(text); + builds.record(&build).await; +} + +/// Keep the builder's `reply` to a build of `agent` from `meta_rev` in `builds`. +async fn record_reply( + builds: &Builds, + agent: &str, + meta_rev: &str, + started_unix: i64, + reply: &Result, +) { + let (outcome, text) = match reply { + Err(why) => (BuildOutcome::Error, why), + Ok(FromMetaReply::Unavailable { reason }) => (BuildOutcome::Error, reason), + Ok(FromMetaReply::Failed { error_tail }) => (BuildOutcome::Failed, error_tail), + Ok(FromMetaReply::Built { out_path, .. }) => (BuildOutcome::Built, out_path), + }; + record_run(builds, agent, meta_rev, started_unix, outcome, text.clone()).await; +} + +/// Build `agent` from `internal/meta`'s head and record the closure. A build +/// the builder answers, or fails to be asked, is also kept in `builds`; one that +/// never got as far as the request (no forge, no head) is not. pub(crate) async fn run( head: Option<&impl MetaRepo>, queue: Option<&impl MetaBuilds>, + builds: &Builds, agent: String, pr: Option, ) -> Outcome { @@ -297,7 +354,10 @@ pub(crate) async fn run( Ok(request) => request, Err(e) => return no_verdict(format!("encoding the build request: {e}")), }; - match queue.build(request).await { + let started_unix = unix_now(); + let reply = queue.build(request).await; + record_reply(builds, &agent, &meta_rev, started_unix, &reply).await; + match reply { Err(why) => no_verdict(why), Ok(FromMetaReply::Unavailable { reason }) => no_verdict(reason), Ok(FromMetaReply::Failed { error_tail }) => { @@ -334,7 +394,9 @@ pub(crate) async fn run( #[cfg(test)] mod tests { - use std::sync::Mutex; + use std::sync::{Arc, Mutex}; + + use crate::builder::runs::tests::FakeDoc; use super::*; @@ -452,11 +514,68 @@ mod tests { } async fn run_with(head: &FakeHead, queue: &FakeQueue) -> Outcome { - run(Some(head), Some(queue), "atlas".to_owned(), None).await + run_recording(head, queue, &Arc::default()).await + } + + /// [`run_with`], keeping the builds in `doc`. + async fn run_recording(head: &FakeHead, queue: &FakeQueue, doc: &Arc) -> Outcome { + let builds = Builds::Fake(Arc::clone(doc)); + run(Some(head), Some(queue), &builds, "atlas".to_owned(), None).await } async fn run_for_pr(head: &FakeHead, queue: &FakeQueue, pr: u64) -> Outcome { - run(Some(head), Some(queue), "atlas".to_owned(), Some(pr)).await + let builds = Builds::Fake(Arc::default()); + run( + Some(head), + Some(queue), + &builds, + "atlas".to_owned(), + Some(pr), + ) + .await + } + + /// The builds page lists these beside the config PRs' builds: which agent, + /// at which `internal/meta` commit, how it ended, and with what. + #[tokio::test] + async fn every_build_the_builder_answers_is_kept_for_the_builds_page() { + let head = FakeHead::reading(Ok(Some(REV))); + let green = Arc::new(FakeDoc::default()); + run_recording(&head, &FakeQueue::built(), &green).await; + let builds = green.builds(); + let [built] = builds.as_slice() else { + panic!("one build kept: {builds:?}") + }; + assert_eq!(built.repo, "internal/meta"); + assert_eq!(built.pr, None); + assert_eq!(built.agent.as_deref(), Some("atlas")); + assert_eq!(built.head_sha, REV); + assert_eq!(built.outcome, BuildOutcome::Built); + assert_eq!(built.out_path.as_deref(), Some("/nix/store/x-toplevel")); + assert!( + built.id.starts_with(&format!("meta-atlas-{}-", &REV[..12])), + "{}", + built.id + ); + + let red = Arc::new(FakeDoc::default()); + run_recording(&head, &nix_failed("error: boom"), &red).await; + let builds = red.builds(); + let [failed] = builds.as_slice() else { + panic!("one build kept: {builds:?}") + }; + assert_eq!(failed.outcome, BuildOutcome::Failed); + assert_eq!(failed.error_tail.as_deref(), Some("error: boom")); + + let none = Arc::new(FakeDoc::default()); + let silent = FakeQueue::answering(Err("no builder is listening".to_owned())); + run_recording(&head, &silent, &none).await; + let builds = none.builds(); + let [error] = builds.as_slice() else { + panic!("one build kept: {builds:?}") + }; + assert_eq!(error.outcome, BuildOutcome::Error); + assert_eq!(error.detail.as_deref(), Some("no builder is listening")); } #[tokio::test] @@ -586,13 +705,16 @@ mod tests { ("no queue", Some(FakeHead::reading(Ok(Some(REV)))), false), ] { let queue = FakeQueue::built(); + let doc = Arc::new(FakeDoc::default()); let outcome = run( head.as_ref(), with_queue.then_some(&queue), + &Builds::Fake(Arc::clone(&doc)), "atlas".to_owned(), None, ) .await; + assert!(doc.builds().is_empty(), "{name}: nothing was built"); let Outcome::Failed(error) = outcome else { panic!("{name}: not failed") };