vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

the builds list keeps the swarm's deploy builds too

Browse source 
build_meta, the build of each agent from internal/meta before a deploy, now
writes its result into the same list as the config PR builds: the agent, the
internal/meta commit, how it ended and what nix printed. The builds card titles
them by agent and shows them with the PR builds.
This commit is contained in:
iris 2026-10-30 02:06:07 +01:00
commit b74011bb54
9 changed files with 212 additions and 33 deletions
Download patch file Download diff file Expand all files Collapse all files

20
docs/swarm/builder.md
View file

@ -61,8 +61,9 @@ Where it runs and how it's enabled: [`services.md`](services.md#builder).
## Build records
`swarm-controller` records each build when it writes the final status:
built, failed, or no build ran or came back. The newest 100 are one JSON
`swarm-controller` records each config PR build when it writes the final status:
built, failed, or no build ran or came back. It also records each [deploy-gate
build](#deploy-gate) the builder answers, in the same list. The newest 100 are one JSON
array under the key `recent` in the `builder-runs` KV bucket on the swarm
queue, which only the controller reads or writes. It creates the bucket on
first use and adds each build with a compare-and-set write, so the records
@ -72,8 +73,12 @@ toward the 100.
Each record has:
- `id`: `<agent>-<first 12 characters of the commit>-<started_unix>`
- `repo` (`agent-configs/<agent>`), `pr` and `head_sha`
- `id`: `<agent>-<first 12 characters of the commit>-<started_unix>` for a
config PR's build, `meta-<agent>-<first 12 characters of the commit>-<started_unix>`
for a deploy's
- `repo` (`agent-configs/<agent>`), `pr` and `head_sha`; a deploy's build has
`repo` `internal/meta`, no `pr`, the `internal/meta` commit as `head_sha`,
and `agent`, the agent it built
- `started_unix` and `finished_unix`
- `outcome`: `built`, `failed` or `error`, the status's `success`, `failure`
or `error`
@ -86,6 +91,13 @@ newest 100. Both are behind the swarm UI vhost's `admins` gate, and answer 503
when the controller has no swarm queue or can't open the bucket, 500 when it
can't read it.
The controller records a deploy-gate build when the builder answers it, or when
asking it fails (no builder listening, no reply). A build that never got as far
as a request (no forge access, no `internal/meta` head) leaves no record, so a
controller that can't reach the forge doesn't fill the list with retries. Every
ten-minute retry of a failing agent is a record, and a version bump records one
build per agent, so the 100 cover fewer days when builds fail.
The final status links to `https://<swarm domain>/jobs?run=<id>`, the swarm
UI's jobs tab, where the builds are a card under the job graph. Selecting
**Details** on a red `swarm-builder` check opens that build's card, with `nix`'s tail, in the browser. The raw record

10
docs/swarm/ui.md
View file

@ -27,7 +27,7 @@ shut. A dropdown inside a dialog opens above the dialog and the dialog doesn't c
| `/agents/<name>/terminal` | one agent's live terminal |
| `/agents/<name>/subagents/<subagent>/terminal` | one of that agent's subagents' live terminal |
| `/reports` | two folded cards, each read only once opened: the **alerts** the swarm's Grafana holds now, the firing ones, with a button for the pending, no-data and error ones and a link to Grafana's own alert list; read through the controller with a Viewer service-account token, and a cross-repo **issue report** |
| `/jobs` | the controller's job graph, where agent creation and credential mints show progress, and under it the swarm builder's recent builds of agent config PRs, with what nix printed when one failed, and the review classifier's recent runs, a try-out on a real PR, and its settings |
| `/jobs` | the controller's job graph, where agent creation and credential mints show progress, and under it the swarm builder's recent builds (agent config PRs, and agents built from `internal/meta` before a deploy), with what nix printed when one failed, and the review classifier's recent runs, a try-out on a real PR, and its settings |
| `/messages` | the swarm-wide message feed: every message between agents, and to them from the operator, reminders and compaction, with a history window and a filter on sender or recipient; a folded **broadcast** card above the feed sends one message to more than one agent at once, and a folded **schedules** card holds the scheduled prompts: add, edit, pause, resume and remove |
| `/notifications` | the notification router's flows: a switch for each built-in, every flow drawn as a graph, and an editor for custom flows |
@ -498,9 +498,11 @@ shut, and each reads nothing until you open it; a link to
a build opens the builds card. The old `/builds` address redirects there and
keeps its query.
The **builds** card lists the agent config PR heads the swarm builder built,
newest first, from `GET /api/builder/runs`. Each card shows the repo and PR,
the first seven characters of the commit, how the build ended (**built**,
The **builds** card lists what the swarm builder built, newest first, from
`GET /api/builder/runs`: agent config PR heads, and the agents it builds from
`internal/meta` before each deploy (a version bump, the pin fan-out or a merged
config PR). Each card shows the repo and PR, or for a deploy's build the agent
and `internal/meta`, the first seven characters of the commit, how the build ended (**built**,
**failed** or **error**), when it started and how long it took. Select a card to
read it: for a failed build, what `nix` printed; for a built one, the system
closure; for an error, why no build came back.

19
frontend/packages/swarm-ui/src/pages/builds/BuildsPage.tsx
View file

@ -1,6 +1,6 @@
// <BuildsCard> — the swarm builder's recent builds, newest first: which agent
// config PR heads it built, how each ended, and what `nix` printed when one
// failed. One card per build; selecting a card opens it. Reads
// <BuildsCard> — the swarm builder's recent builds, newest first: the agent
// config PR heads it built, and the agents it built from `internal/meta` before
// a deploy, how each ended, and what `nix` printed when one failed. One card per build; selecting a card opens it. Reads
// `GET /api/builder/runs`. A `swarm-builder` commit status links to the jobs
// page as `/jobs?run=<id>`, which opens that build's card and scrolls to it.
// The card sits under the job graph on that page; the old `/builds` address
@ -22,6 +22,7 @@ import {
OUTCOME_LABEL,
OUTCOME_TONE,
bodyText,
runTitle,
wantedRun,
type BuildRun,
} from "./builds-types.js";
@ -56,9 +57,7 @@ function BuildCard({
onClick={onToggle}
>
<span class="ui-builds-title">
<strong>
{run.repo} #{run.pr}
</strong>
<strong>{runTitle(run)}</strong>
<code>{run.head_sha.slice(0, 7)}</code>
</span>
<Badge
@ -138,9 +137,9 @@ function BuildsBody() {
/>
</div>
<p>
The agent config PR heads the swarm builder built, newest first: how
each ended, and what nix printed when one failed. Select a build to read
it.
What the swarm builder built, newest first: agent config PR heads, and
the agents it builds from internal/meta before a deploy. How each ended,
and what nix printed when one failed. Select a build to read it.
</p>
{error ? (
<ApiErrorPanel
@ -178,7 +177,7 @@ export function BuildsCard() {
<LazyDisclosure
title="builds"
icon="🔨"
hint="the agent config PRs the swarm builder built"
hint="what the swarm builder built: agent config PRs and deploys"
startOpen={wantedRun(useSearch()) !== null}
>
<BuildsBody />

17
frontend/packages/swarm-ui/src/pages/builds/builds-types.ts
View file

@ -10,9 +10,12 @@ export type BuildOutcome = "built" | "failed" | "error";
export interface BuildRun {
/** `<agent>-<first 12 of head_sha>-<started_unix>`. */
id: string;
/** `agent-configs/<agent>`. */
/** `agent-configs/<agent>` for a config PR's build, `internal/meta` for a deploy's. */
repo: string;
pr: number;
/** The config PR a build checks; a deploy's build has none. */
pr?: number | null;
/** The agent a deploy's build built; a PR build's is in `repo`. */
agent?: string | null;
head_sha: string;
started_unix: number;
finished_unix: number;
@ -37,6 +40,16 @@ export const OUTCOME_TONE: Record<BuildOutcome, BadgeTone> = {
error: "warning",
};
/**
* A run's card title: `agent-configs/atlas #3` for a config PR's build,
* `atlas from internal/meta` for a deploy's (the swarm builds every agent from
* `internal/meta` before sending its deploy).
*/
export function runTitle(run: BuildRun): string {
if (run.pr !== undefined && run.pr !== null) return `${run.repo} #${run.pr}`;
return run.agent ? `${run.agent} from ${run.repo}` : run.repo;
}
/** The run a `?run=<id>` query names, or `null` when it names none. */
export function wantedRun(search: string): string | null {
const id = new URLSearchParams(search).get("run");

23
frontend/packages/swarm-ui/test/builds-types.test.mjs
View file

@ -3,7 +3,11 @@
// stripping, so the `.ts` module is imported as it is.
import assert from "node:assert/strict";
import { test } from "node:test";
import { bodyText, wantedRun } from "../src/pages/builds/builds-types.ts";
import {
bodyText,
runTitle,
wantedRun,
} from "../src/pages/builds/builds-types.ts";
const base = {
id: "atlas-abc-1",
@ -49,3 +53,20 @@ test("a run with no text for its outcome shows no body", () => {
assert.equal(bodyText({ ...base, outcome: "failed" }), null);
assert.equal(bodyText({ ...base, outcome: "built", error_tail: "x" }), null);
});
test("a config PR's build is titled by its repo and PR, a deploy's by its agent", () => {
assert.equal(
runTitle({ ...base, outcome: "built" }),
"agent-configs/atlas #3",
);
const deploy = {
...base,
repo: "internal/meta",
pr: undefined,
agent: "atlas",
outcome: "built",
};
assert.equal(runTitle(deploy), "atlas from internal/meta");
assert.equal(runTitle({ ...deploy, pr: null }), "atlas from internal/meta");
assert.equal(runTitle({ ...deploy, agent: null }), "internal/meta");
});

5
swarm-controller/src/builder.rs
View file

@ -193,7 +193,8 @@ async fn record(
job.rev.get(..12).unwrap_or(&job.rev)
),
repo: format!("{CONFIG_ORG}/{}", job.agent),
pr: job.pr,
pr: Some(job.pr),
agent: None,
head_sha: job.rev.clone(),
started_unix,
finished_unix: unix_now(),
@ -343,7 +344,7 @@ mod tests {
panic!("one build recorded: {builds:?}")
};
assert_eq!(build.repo, "agent-configs/atlas");
assert_eq!((build.pr, build.head_sha.as_str()), (3, "abc"));
assert_eq!((build.pr, build.head_sha.as_str()), (Some(3), "abc"));
assert_eq!(build.outcome, BuildOutcome::Failed);
assert_eq!(build.error_tail.as_deref(), Some(tail));
assert_eq!((&build.out_path, &build.detail), (&None, &None));

16
swarm-controller/src/builder/runs.rs
View file

@ -51,10 +51,17 @@ pub(crate) struct BuildRecord {
/// `<agent>-<first 12 of head_sha>-<started_unix>`, the last segment
/// of the link on the build's status.
pub(crate) id: String,
/// `agent-configs/<agent>`.
/// `agent-configs/<agent>` for a config PR's build, `internal/meta` for a
/// deploy's (`meta_build`).
pub(crate) repo: String,
pub(crate) pr: u64,
/// The commit built.
/// The config PR a build checks. A `meta_build` build has none.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub(crate) pr: Option<u64>,
/// The agent a `meta_build` build built; a PR build's is in `repo`.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub(crate) agent: Option<String>,
/// The commit built: the PR's head, or `internal/meta`'s for a deploy's
/// build.
pub(crate) head_sha: String,
/// Unix seconds: when the node sent the build, and when its answer came.
pub(crate) started_unix: i64,
@ -362,7 +369,8 @@ pub(crate) mod tests {
BuildRecord {
id: id.to_owned(),
repo: "agent-configs/atlas".to_owned(),
pr: 3,
pr: Some(3),
agent: None,
head_sha: "abc".to_owned(),
started_unix: 1_000,
finished_unix: 1_060,

3
swarm-controller/src/jobq.rs
View file

@ -655,7 +655,8 @@ async fn build_meta(
agent: String,
pr: Option<u64>,
) -> hive_jobq::scheduler::Outcome {
meta_build::run(forge.as_deref(), deps.queue.as_ref(), agent, pr).await
let builds = builder::runs::Builds::Live(deps.queue.clone());
meta_build::run(forge.as_deref(), deps.queue.as_ref(), &builds, agent, pr).await
}
/// What an `UpdateMeta` pass decided: the deploys to send, and the placements

132
swarm-controller/src/meta_build.rs
View file

@ -35,6 +35,7 @@ use swarm_wire::agent_closures::AgentClosure;
use swarm_wire::builder::{BuildFromMeta, FromMetaReply, META_SUBJECT, REPLY_DEADLINE};
use crate::agent_renewal::unix_now;
use crate::builder::runs::{BuildOutcome, BuildRecord, Builds};
use crate::forge::{self, objects::SHARED_ORG};
use crate::meta_flake::{META_REPO, MetaForge};
@ -263,10 +264,66 @@ async fn clear_failure(head: &impl MetaRepo, pr: u64, agent: &str, meta_rev: &st
}
}
/// Build `agent` from `internal/meta`'s head and record the closure.
/// Store the build of `agent` from `meta_rev` that ended as `outcome`, `text`
/// being its store path, `nix`'s tail or the reason, in the list the builds
/// page shows beside the config PRs' builds. A failed write is a warning in
/// `Builds::record` and changes nothing about the node.
async fn record_run(
builds: &Builds,
agent: &str,
meta_rev: &str,
started_unix: i64,
outcome: BuildOutcome,
text: String,
) {
let mut build = BuildRecord {
id: format!(
"meta-{agent}-{}-{started_unix}",
meta_rev.get(..12).unwrap_or(meta_rev)
),
repo: format!("{SHARED_ORG}/{META_REPO}"),
pr: None,
agent: Some(agent.to_owned()),
head_sha: meta_rev.to_owned(),
started_unix,
finished_unix: unix_now(),
outcome,
out_path: None,
error_tail: None,
detail: None,
};
*match outcome {
BuildOutcome::Built => &mut build.out_path,
BuildOutcome::Failed => &mut build.error_tail,
BuildOutcome::Error => &mut build.detail,
} = Some(text);
builds.record(&build).await;
}
/// Keep the builder's `reply` to a build of `agent` from `meta_rev` in `builds`.
async fn record_reply(
builds: &Builds,
agent: &str,
meta_rev: &str,
started_unix: i64,
reply: &Result<FromMetaReply, String>,
) {
let (outcome, text) = match reply {
Err(why) => (BuildOutcome::Error, why),
Ok(FromMetaReply::Unavailable { reason }) => (BuildOutcome::Error, reason),
Ok(FromMetaReply::Failed { error_tail }) => (BuildOutcome::Failed, error_tail),
Ok(FromMetaReply::Built { out_path, .. }) => (BuildOutcome::Built, out_path),
};
record_run(builds, agent, meta_rev, started_unix, outcome, text.clone()).await;
}
/// Build `agent` from `internal/meta`'s head and record the closure. A build
/// the builder answers, or fails to be asked, is also kept in `builds`; one that
/// never got as far as the request (no forge, no head) is not.
pub(crate) async fn run(
head: Option<&impl MetaRepo>,
queue: Option<&impl MetaBuilds>,
builds: &Builds,
agent: String,
pr: Option<u64>,
) -> Outcome {
@ -297,7 +354,10 @@ pub(crate) async fn run(
Ok(request) => request,
Err(e) => return no_verdict(format!("encoding the build request: {e}")),
};
match queue.build(request).await {
let started_unix = unix_now();
let reply = queue.build(request).await;
record_reply(builds, &agent, &meta_rev, started_unix, &reply).await;
match reply {
Err(why) => no_verdict(why),
Ok(FromMetaReply::Unavailable { reason }) => no_verdict(reason),
Ok(FromMetaReply::Failed { error_tail }) => {
@ -334,7 +394,9 @@ pub(crate) async fn run(
#[cfg(test)]
mod tests {
use std::sync::Mutex;
use std::sync::{Arc, Mutex};
use crate::builder::runs::tests::FakeDoc;
use super::*;
@ -452,11 +514,68 @@ mod tests {
}
async fn run_with(head: &FakeHead, queue: &FakeQueue) -> Outcome {
run(Some(head), Some(queue), "atlas".to_owned(), None).await
run_recording(head, queue, &Arc::default()).await
}
/// [`run_with`], keeping the builds in `doc`.
async fn run_recording(head: &FakeHead, queue: &FakeQueue, doc: &Arc<FakeDoc>) -> Outcome {
let builds = Builds::Fake(Arc::clone(doc));
run(Some(head), Some(queue), &builds, "atlas".to_owned(), None).await
}
async fn run_for_pr(head: &FakeHead, queue: &FakeQueue, pr: u64) -> Outcome {
run(Some(head), Some(queue), "atlas".to_owned(), Some(pr)).await
let builds = Builds::Fake(Arc::default());
run(
Some(head),
Some(queue),
&builds,
"atlas".to_owned(),
Some(pr),
)
.await
}
/// The builds page lists these beside the config PRs' builds: which agent,
/// at which `internal/meta` commit, how it ended, and with what.
#[tokio::test]
async fn every_build_the_builder_answers_is_kept_for_the_builds_page() {
let head = FakeHead::reading(Ok(Some(REV)));
let green = Arc::new(FakeDoc::default());
run_recording(&head, &FakeQueue::built(), &green).await;
let builds = green.builds();
let [built] = builds.as_slice() else {
panic!("one build kept: {builds:?}")
};
assert_eq!(built.repo, "internal/meta");
assert_eq!(built.pr, None);
assert_eq!(built.agent.as_deref(), Some("atlas"));
assert_eq!(built.head_sha, REV);
assert_eq!(built.outcome, BuildOutcome::Built);
assert_eq!(built.out_path.as_deref(), Some("/nix/store/x-toplevel"));
assert!(
built.id.starts_with(&format!("meta-atlas-{}-", &REV[..12])),
"{}",
built.id
);
let red = Arc::new(FakeDoc::default());
run_recording(&head, &nix_failed("error: boom"), &red).await;
let builds = red.builds();
let [failed] = builds.as_slice() else {
panic!("one build kept: {builds:?}")
};
assert_eq!(failed.outcome, BuildOutcome::Failed);
assert_eq!(failed.error_tail.as_deref(), Some("error: boom"));
let none = Arc::new(FakeDoc::default());
let silent = FakeQueue::answering(Err("no builder is listening".to_owned()));
run_recording(&head, &silent, &none).await;
let builds = none.builds();
let [error] = builds.as_slice() else {
panic!("one build kept: {builds:?}")
};
assert_eq!(error.outcome, BuildOutcome::Error);
assert_eq!(error.detail.as_deref(), Some("no builder is listening"));
}
#[tokio::test]
@ -586,13 +705,16 @@ mod tests {
("no queue", Some(FakeHead::reading(Ok(Some(REV)))), false),
] {
let queue = FakeQueue::built();
let doc = Arc::new(FakeDoc::default());
let outcome = run(
head.as_ref(),
with_queue.then_some(&queue),
&Builds::Fake(Arc::clone(&doc)),
"atlas".to_owned(),
None,
)
.await;
assert!(doc.builds().is_empty(), "{name}: nothing was built");
let Outcome::Failed(error) = outcome else {
panic!("{name}: not failed")
};