swarm-builder: build each agent config PR head and post a commit status

A new `swarm-builder` service, in its own container on the host that sets
`deploy.builder.enable` (defaulted on by `allSwarmServices` where the host
has the builder's store leaf, like the review classifier).

When an `agent-configs/<agent>` PR is opened, reopened or synchronized,
swarm-controller queues a `build_agent` jobq node. The node sets a
`swarm-builder` commit status to pending on the PR head, sends
`{agent, rev, pr}` to the builder over NATS request-reply
(`$SWARM.builder.build`), and writes success (store path), failure (last
line nix printed) or error (no builder / no reply) on the head. The node
fails when the build does, so the Jobs page matches the PR.

The builder builds nix/builder/check-flake's `check` with every input
overridden: hyperhive and nixpkgs to the host's own
`c0re.hyperhiveFlake`/`c0re.nixpkgsFlake` (what a hive there renders into
its meta flake), and the agent to the PR head on the forge. Evaluation is
pure and runs in the container; builds go to the host nix daemon as an
untrusted client. The private config repo is fetched with the token of a
new read-only `builder` forge account (in the `hives` team), which the
controller mints into `swarm/services/builder/forge/token`; git gets it
from a credential helper reading the child's environment, so it is in no
argument or URL.

Credentials: a `builder` store leaf minted by glue-bao-tls on the store
host, a `swarm-builder` reader policy (forge token + its own OIDC client
secret), an authelia machine client, and a `--builder-client` grant in the
queue's auth-callout responder. The controller gains a grant on the token
path and the secret publisher a deny on it.

This is the agent-config build check atlas scoped on #5081 (slice 3):
nothing is pushed to a cache, nothing is written to KV, and a red status
gates no deploy. The check composes agent-base + the config without the
values a hive adds (CA, docs, claude-code path, ports, input follows), so
it can disagree with a hive's own build.

swarm-controller/src/builder.rs

@@ -0,0 +1,304 @@
+//! The `BuildAgent` node: send a config PR's head to the swarm builder over the
+//! queue, and turn its answer into the [`CONTEXT`] commit status on that head.
+//!
+//! The status is the check's only record besides the node itself: nothing is
+//! pushed, and the closure the builder made stays in its host's store. A red
+//! status blocks nothing; the merge deploys as before.
+
+use std::future::Future;
+
+use forgejo_api::structs::CommitStatusState;
+use hive_jobq::scheduler::Outcome;
+use swarm_queue_client::builder::{BuildAgent, REPLY_DEADLINE, Reply, SUBJECT};
+
+use crate::forge::{self, CONFIG_ORG};
+
+/// The commit status context the check writes.
+pub const CONTEXT: &str = "swarm-builder";
+
+/// Where a build is sent: the queue in production, a fake in the tests.
+/// `Err` is a build that never reached a builder or never came back.
+pub(crate) trait BuildSink {
+    fn build(&self, request: Vec<u8>) -> impl Future<Output = Result<Reply, String>> + Send;
+}
+
+impl BuildSink for async_nats::Client {
+    async fn build(&self, request: Vec<u8>) -> Result<Reply, String> {
+        use async_nats::RequestErrorKind;
+
+        let sent = async_nats::Request::new()
+            .payload(request.into())
+            .timeout(Some(REPLY_DEADLINE));
+        match self.send_request(SUBJECT, sent).await {
+            Ok(msg) => serde_json::from_slice(&msg.payload)
+                .map_err(|e| format!("the builder's reply did not decode: {e}")),
+            Err(e) => Err(match e.kind() {
+                RequestErrorKind::NoResponders => "no swarm builder is listening: \
+                     deploy.builder.enable is set on no host, or its service is not connected \
+                     to the swarm queue"
+                    .to_owned(),
+                RequestErrorKind::TimedOut => format!(
+                    "the swarm builder sent no answer within {} minutes",
+                    REPLY_DEADLINE.as_secs() / 60
+                ),
+                _ => format!("sending the build to the swarm builder failed: {e}"),
+            }),
+        }
+    }
+}
+
+/// Where a status is written: the forge in production, a fake in the tests.
+pub(crate) trait StatusSink {
+    fn set(
+        &self,
+        agent: &str,
+        rev: &str,
+        state: CommitStatusState,
+        description: &str,
+    ) -> impl Future<Output = anyhow::Result<()>> + Send;
+}
+
+impl StatusSink for forge::Client {
+    async fn set(
+        &self,
+        agent: &str,
+        rev: &str,
+        state: CommitStatusState,
+        description: &str,
+    ) -> anyhow::Result<()> {
+        forge::within_forge_deadline(
+            "build_agent status",
+            self.create_commit_status(CONFIG_ORG, agent, rev, CONTEXT, state, description),
+        )
+        .await
+        .unwrap_or_else(|| Err(anyhow::anyhow!("no answer from the forge")))
+    }
+}
+
+/// Build `job` and write its status. The node fails when the build does, so
+/// the Jobs page shows red where the PR does.
+pub(crate) async fn run(
+    queue: Option<&impl BuildSink>,
+    statuses: Option<&impl StatusSink>,
+    job: BuildAgent,
+) -> Outcome {
+    let Some(statuses) = statuses else {
+        return Outcome::Failed(
+            "no forge access here (SWARM_CONTROLLER_FORGE_URL unset, or its token not in the \
+             swarm secret store), so no build status can be written"
+                .to_owned(),
+        );
+    };
+    let set = |state, description: String| {
+        let job = &job;
+        async move {
+            statuses
+                .set(&job.agent, &job.rev, state, &description)
+                .await
+                .map_err(|e| format!("{e:#}"))
+        }
+    };
+    let Some(queue) = queue else {
+        let error = "no swarm queue is configured on this host, so no build can be sent";
+        return settle(set(CommitStatusState::Error, error.to_owned()).await, error);
+    };
+    if let Err(e) = set(
+        CommitStatusState::Pending,
+        "building on the swarm builder".to_owned(),
+    )
+    .await
+    {
+        tracing::warn!(agent = %job.agent, rev = %job.rev, error = %e, "build_agent: the pending status was not written");
+    }
+    let request = match serde_json::to_vec(&job) {
+        Ok(request) => request,
+        Err(e) => return Outcome::Failed(format!("encoding the build: {e}")),
+    };
+    match queue.build(request).await {
+        Ok(Reply::Built { out_path }) => {
+            tracing::info!(agent = %job.agent, rev = %job.rev, %out_path, "build_agent: built");
+            match set(CommitStatusState::Success, out_path).await {
+                Ok(()) => Outcome::Done,
+                Err(e) => Outcome::Failed(format!("built, but the status was not written: {e}")),
+            }
+        }
+        Ok(Reply::Failed { error_tail }) => {
+            let last = last_line(&error_tail);
+            settle(
+                set(CommitStatusState::Failure, last.to_owned()).await,
+                &format!("the build failed: {error_tail}"),
+            )
+        }
+        Err(error) => settle(set(CommitStatusState::Error, error.clone()).await, &error),
+    }
+}
+
+/// A failed node carrying `error`, and the status write's own failure when
+/// there was one.
+fn settle(written: Result<(), String>, error: &str) -> Outcome {
+    match written {
+        Ok(()) => Outcome::Failed(error.to_owned()),
+        Err(e) => Outcome::Failed(format!("{error} (and the status was not written: {e})")),
+    }
+}
+
+/// The last non-blank line of `tail`, which for `nix` is the error itself.
+fn last_line(tail: &str) -> &str {
+    tail.lines()
+        .rev()
+        .map(str::trim)
+        .find(|l| !l.is_empty())
+        .unwrap_or("the build failed")
+}
+
+#[cfg(test)]
+mod tests {
+    use std::sync::Mutex;
+
+    use super::*;
+
+    struct FakeQueue(Result<Reply, String>);
+
+    impl BuildSink for FakeQueue {
+        async fn build(&self, request: Vec<u8>) -> Result<Reply, String> {
+            let sent: BuildAgent = serde_json::from_slice(&request).expect("a build request");
+            assert_eq!(sent, job());
+            self.0.clone()
+        }
+    }
+
+    #[derive(Default)]
+    struct FakeStatuses(Mutex<Vec<(CommitStatusState, String)>>);
+
+    impl StatusSink for FakeStatuses {
+        async fn set(
+            &self,
+            agent: &str,
+            rev: &str,
+            state: CommitStatusState,
+            description: &str,
+        ) -> anyhow::Result<()> {
+            assert_eq!((agent, rev), ("atlas", "abc"));
+            self.0.lock().unwrap().push((state, description.to_owned()));
+            Ok(())
+        }
+    }
+
+    fn job() -> BuildAgent {
+        BuildAgent {
+            agent: "atlas".to_owned(),
+            rev: "abc".to_owned(),
+            pr: 3,
+        }
+    }
+
+    async fn statuses_after(queue: Option<FakeQueue>) -> (Outcome, Vec<CommitStatusState>) {
+        let statuses = FakeStatuses::default();
+        let outcome = run(queue.as_ref(), Some(&statuses), job()).await;
+        let states = statuses.0.into_inner().unwrap();
+        (outcome, states.into_iter().map(|(s, _)| s).collect())
+    }
+
+    #[tokio::test]
+    async fn a_built_config_goes_pending_then_green() {
+        let (outcome, states) = statuses_after(Some(FakeQueue(Ok(Reply::Built {
+            out_path: "/nix/store/x".to_owned(),
+        }))))
+        .await;
+        assert!(matches!(outcome, Outcome::Done));
+        assert_eq!(
+            states,
+            [CommitStatusState::Pending, CommitStatusState::Success]
+        );
+    }
+
+    #[tokio::test]
+    async fn a_failed_build_goes_red_with_its_last_line_and_fails_the_node() {
+        let statuses = FakeStatuses::default();
+        let queue = FakeQueue(Ok(Reply::Failed {
+            error_tail: "evaluating\nerror: boom\n".to_owned(),
+        }));
+        let outcome = run(Some(&queue), Some(&statuses), job()).await;
+        assert!(matches!(&outcome, Outcome::Failed(e) if e.contains("error: boom")));
+        let written = statuses.0.into_inner().unwrap();
+        assert_eq!(
+            written.last(),
+            Some(&(CommitStatusState::Failure, "error: boom".to_owned()))
+        );
+    }
+
+    #[tokio::test]
+    async fn no_builder_is_an_error_status_not_a_failure() {
+        let (outcome, states) =
+            statuses_after(Some(FakeQueue(Err("no swarm builder".to_owned())))).await;
+        assert!(matches!(outcome, Outcome::Failed(_)));
+        assert_eq!(
+            states,
+            [CommitStatusState::Pending, CommitStatusState::Error]
+        );
+    }
+
+    #[tokio::test]
+    async fn no_queue_is_an_error_status() {
+        let (outcome, states) = statuses_after(None).await;
+        assert!(matches!(&outcome, Outcome::Failed(e) if e.contains("no swarm queue")));
+        assert_eq!(states, [CommitStatusState::Error]);
+    }
+
+    #[tokio::test]
+    async fn no_forge_fails_the_node_without_sending_the_build() {
+        struct Unreachable;
+        impl BuildSink for Unreachable {
+            async fn build(&self, _: Vec<u8>) -> Result<Reply, String> {
+                panic!("a build with nowhere to report must not be sent")
+            }
+        }
+        let outcome = run(Some(&Unreachable), None::<&FakeStatuses>, job()).await;
+        assert!(matches!(&outcome, Outcome::Failed(e) if e.contains("no forge access")));
+    }
+
+    /// The queue half against a real server: no builder listening names the
+    /// option, and a listener's reply comes back decoded.
+    #[tokio::test]
+    async fn a_build_reaches_a_listening_builder_and_fails_without_one() {
+        use futures_util::StreamExt as _;
+
+        let nats = crate::test_nats::TestNats::start();
+        let client = nats.client().await;
+        let unanswered = client.build(b"{}".to_vec()).await;
+        assert!(
+            matches!(&unanswered, Err(e) if e.contains("deploy.builder.enable")),
+            "{unanswered:?}"
+        );
+
+        let builder = nats.client().await;
+        let mut requests = builder
+            .queue_subscribe(SUBJECT, swarm_queue_client::builder::QUEUE_GROUP.to_owned())
+            .await
+            .expect("subscribe");
+        // A message the builder sends itself and receives back proves the
+        // server processed the subscription before it.
+        let mut probe = builder
+            .subscribe("builder-test.probe")
+            .await
+            .expect("subscribe");
+        builder
+            .publish("builder-test.probe", "".into())
+            .await
+            .expect("publish");
+        probe.next().await.expect("the probe comes back");
+        let built = Reply::Built {
+            out_path: "/nix/store/x".to_owned(),
+        };
+        let answer = built.clone();
+        tokio::spawn(async move {
+            let msg = requests.next().await.expect("a request");
+            let reply = serde_json::to_vec(&answer).expect("encodes");
+            builder
+                .publish(msg.reply.expect("a reply subject"), reply.into())
+                .await
+                .expect("publish");
+        });
+        assert_eq!(client.build(b"{}".to_vec()).await, Ok(built));
+    }
+}

swarm-controller/src/config_pr.rs

@@ -16,8 +16,10 @@
 //!   instead of waiting up to `POLL_INTERVAL`.
 //!
 //! [`merged`] reads the same delivery for a merge, which the webhook handler
-//! turns into a deploy. The poll has no counterpart: it lists open PRs only,
-//! so a merge whose delivery is lost deploys nothing.
+//! turns into a deploy, and [`changed_head`] reads it for a PR's new head,
+//! which the handler turns into a build check (`crate::builder`). The poll
+//! has no counterpart to either: a merge whose delivery is lost deploys
+//! nothing, and a head whose delivery is lost is not built.
 
 use std::collections::HashMap;
 use std::sync::{Arc, Mutex};
@@ -58,12 +60,17 @@ struct WebhookPullRequest {
     /// hive builds its config repo's `main`.
     #[serde(default)]
     base: Option<WebhookBranch>,
+    /// The PR's branch, whose `sha` is the commit a build check builds.
+    #[serde(default)]
+    head: Option<WebhookBranch>,
 }
 
 #[derive(Deserialize)]
 struct WebhookBranch {
     #[serde(rename = "ref")]
     name: String,
+    #[serde(default)]
+    sha: Option<String>,
 }
 
 /// A config PR that was just merged: whose config, and the commit to deploy.
@@ -104,6 +111,43 @@ pub fn merged(body: &[u8]) -> Option<MergedConfigPr> {
     })
 }
 
+/// A config PR whose head just changed: whose config, the head commit, and
+/// the PR.
+#[derive(Debug, PartialEq, Eq)]
+pub struct ChangedConfigPr {
+    pub agent: String,
+    pub rev: String,
+    pub pr: u64,
+}
+
+/// The new head a verified `ConfigPr` delivery reports, if it reports one: a
+/// PR opened, reopened, or pushed to (`synchronized`). Edits and labels leave
+/// the head as it was and build nothing. A body that does not parse is
+/// `None`; [`ConfigPrCache::apply_webhook_delivery`] already logs it.
+pub fn changed_head(body: &[u8]) -> Option<ChangedConfigPr> {
+    let payload: ConfigPrWebhookPayload = serde_json::from_slice(body).ok()?;
+    if !matches!(
+        payload.action.as_str(),
+        "opened" | "reopened" | "synchronized"
+    ) || payload.pull_request.state != "open"
+    {
+        return None;
+    }
+    let Some(rev) = payload.pull_request.head.and_then(|h| h.sha) else {
+        tracing::warn!(
+            agent = %payload.repository.name,
+            pr = payload.pull_request.number,
+            "config-pr webhook: PR carries no head sha; nothing built"
+        );
+        return None;
+    };
+    Some(ChangedConfigPr {
+        agent: payload.repository.name,
+        rev,
+        pr: payload.pull_request.number,
+    })
+}
+
 #[derive(Deserialize)]
 struct WebhookRepository {
     /// The config repo's name IS the agent's name — same convention
@@ -369,4 +413,42 @@ mod tests {
             .expect("prior entry must survive a bad delivery");
         assert_eq!(status.pr_number, 5);
     }
+
+    fn head_changed(action: &str) -> Vec<u8> {
+        serde_json::json!({
+            "action": action,
+            "pull_request": {
+                "number": 5,
+                "state": "open",
+                "head": { "ref": "fix", "sha": "def456" },
+                "base": { "ref": "main", "sha": "abc123" },
+            },
+            "repository": { "name": "damocles" },
+        })
+        .to_string()
+        .into_bytes()
+    }
+
+    #[test]
+    fn an_opened_reopened_or_pushed_pr_names_its_head() {
+        for action in ["opened", "reopened", "synchronized"] {
+            assert_eq!(
+                super::changed_head(&head_changed(action)),
+                Some(super::ChangedConfigPr {
+                    agent: "damocles".to_owned(),
+                    rev: "def456".to_owned(),
+                    pr: 5,
+                }),
+                "{action}"
+            );
+        }
+    }
+
+    #[test]
+    fn an_edit_a_label_or_a_close_builds_nothing() {
+        for action in ["edited", "label_updated", "closed"] {
+            assert_eq!(super::changed_head(&head_changed(action)), None, "{action}");
+        }
+        assert_eq!(super::changed_head(&closed(true)), None);
+    }
 }

swarm-controller/src/forge.rs

@@ -21,12 +21,12 @@
 use anyhow::{Context, Result};
 use forgejo_api::structs::{
     AddCollaboratorOption, AddCollaboratorOptionPermission, ChangeFileOperation,
-    ChangeFileOperationOperation, ChangeFilesOptions, CreateBranchProtectionOption,
-    CreateHookOption, CreateHookOptionConfig, CreateHookOptionType, CreateRepoOption,
-    CreateUserOption, EditUserOption, GetTreeQuery, GitEntry, IssueListIssuesQuery,
-    IssueListIssuesQueryState, IssueListIssuesQueryType, RepoDownloadPullDiffOrPatchQuery,
-    RepoGetContentsQuery, RepoGetPullRequestCommitsQuery, RepoListPullRequestsQuery,
-    RepoListPullRequestsQueryState, RepoSearchQuery, StateType,
+    ChangeFileOperationOperation, ChangeFilesOptions, CommitStatusState,
+    CreateBranchProtectionOption, CreateHookOption, CreateHookOptionConfig, CreateHookOptionType,
+    CreateRepoOption, CreateStatusOption, CreateUserOption, EditUserOption, GetTreeQuery, GitEntry,
+    IssueListIssuesQuery, IssueListIssuesQueryState, IssueListIssuesQueryType,
+    RepoDownloadPullDiffOrPatchQuery, RepoGetContentsQuery, RepoGetPullRequestCommitsQuery,
+    RepoListPullRequestsQuery, RepoListPullRequestsQueryState, RepoSearchQuery, StateType,
 };
 use forgejo_api::{ApiErrorKind, Auth, Forgejo, ForgejoError};
 use futures_util::{StreamExt as _, TryStreamExt as _};
@@ -40,6 +40,7 @@ use crate::webhook::DeliveryKind;
 use swarm_queue_client::review_classifier::Commit;
 
 pub mod agent_token;
+pub mod builder_account;
 pub mod classifier_account;
 pub mod docs_grant;
 pub mod hive_token;
@@ -48,6 +49,24 @@ pub mod objects;
 pub mod runner_token;
 pub mod site_admin;
 
+/// The longest commit status description [`Client::create_commit_status`]
+/// sends, in characters. The forge keeps more; its PR view shows one line.
+pub const STATUS_DESCRIPTION_MAX: usize = 200;
+
+/// `description` cut to [`STATUS_DESCRIPTION_MAX`] characters, an ellipsis
+/// marking a cut.
+fn status_description(description: &str) -> String {
+    if description.chars().count() <= STATUS_DESCRIPTION_MAX {
+        return description.to_owned();
+    }
+    let mut cut: String = description
+        .chars()
+        .take(STATUS_DESCRIPTION_MAX - 1)
+        .collect();
+    cut.push('…');
+    cut
+}
+
 /// An agent's open config-PR, as [`Client::list_open_config_prs`] reports it
 /// and `GET /api/agents/{name}/config-pr` serves it.
 #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, ToSchema)]
@@ -163,7 +182,7 @@ const OPERATORS_TEAM: &str = "operators";
 /// The merge gate survives the distinction: [`objects`] provisions the
 /// `operators` team in **both** orgs precisely so branch protection can be
 /// applied in either.
-const CONFIG_ORG: &str = "agent-configs";
+pub(crate) const CONFIG_ORG: &str = "agent-configs";
 
 /// The hive-wide knowledge repo, where the `push` hook lives. Same values as
 /// `hive-c0re::workers::knowledge::{ORG, REPO}`.
@@ -1127,6 +1146,38 @@ impl Client {
         Ok(rows.into_iter().flatten().collect())
     }
 
+    /// Set the commit status named `context` on `sha` of `owner/repo` to
+    /// `state`, with `description` cut to [`STATUS_DESCRIPTION_MAX`]
+    /// characters.
+    ///
+    /// # Errors
+    /// The forge refused the status.
+    pub async fn create_commit_status(
+        &self,
+        owner: &str,
+        repo: &str,
+        sha: &str,
+        context: &str,
+        state: CommitStatusState,
+        description: &str,
+    ) -> Result<()> {
+        self.api
+            .repo_create_status(
+                owner,
+                repo,
+                sha,
+                CreateStatusOption {
+                    context: Some(context.to_owned()),
+                    description: Some(status_description(description)),
+                    state: Some(state),
+                    target_url: None,
+                },
+            )
+            .await
+            .with_context(|| format!("set the {context} status on {sha} of {owner}/{repo}"))?;
+        Ok(())
+    }
+
     /// What the review classifier reads of `owner/repo`'s PR `number`: its
     /// head commit, its commits oldest first with their full messages, and
     /// its unified diff.
@@ -2368,4 +2419,41 @@ mod tests {
             .unwrap_err();
         assert!(!no_repo(&tree), "{tree:#}");
     }
+
+    #[test]
+    fn a_long_status_description_is_cut_with_an_ellipsis() {
+        assert_eq!(status_description("built"), "built");
+        let long = "é".repeat(STATUS_DESCRIPTION_MAX + 5);
+        let cut = status_description(&long);
+        assert_eq!(cut.chars().count(), STATUS_DESCRIPTION_MAX);
+        assert!(cut.ends_with('…'));
+    }
+
+    #[tokio::test]
+    async fn a_commit_status_is_posted_on_the_commit() {
+        let (client, seen) = stub_forge(
+            (201, r#"{"id":1}"#),
+            (404, r#"{"message":"not stubbed"}"#),
+            (404, r#"{"message":"not stubbed"}"#),
+        )
+        .await;
+        client
+            .create_commit_status(
+                CONFIG_ORG,
+                "atlas",
+                "abc",
+                "swarm-builder",
+                CommitStatusState::Failure,
+                "error: boom",
+            )
+            .await
+            .expect("the stub accepts the status");
+        let seen = seen.lock().unwrap();
+        let (method, path, body) = seen.first().expect("one request");
+        assert_eq!(method, "POST");
+        assert_eq!(path, "/api/v1/repos/agent-configs/atlas/statuses/abc");
+        assert_eq!(body["context"], "swarm-builder");
+        assert_eq!(body["state"], "failure");
+        assert_eq!(body["description"], "error: boom");
+    }
 }

swarm-controller/src/forge/builder_account.rs

@@ -0,0 +1,284 @@
+//! The swarm builder's read-only forge account, [`LOGIN`]: created here with
+//! the admin API, put in the `hives` team of `agent-configs` (read on every
+//! config repo, including ones created later), and given one token, stored
+//! at `swarm_secret_client::builder::forge_token_path`, where the builder
+//! reads it under its own store identity to fetch the config it builds.
+//!
+//! The builder never writes to the forge: its build results become commit
+//! statuses through this daemon's own client (`crate::builder`).
+//!
+//! The token decision is [`super::agent_token`]'s, under this module's name
+//! and scopes, so a stored token the forge still lists with the same last
+//! eight characters and scopes is kept, and a pass whose reads fail queues
+//! nothing.
+
+use std::sync::Arc;
+
+use anyhow::{Context, Result, bail};
+use forgejo_api::structs::RepoSearchQuery;
+use forgejo_api::{Auth, Forgejo};
+use swarm_secret_client::builder::forge_token_path;
+use swarm_secret_client::forge;
+
+use super::Client;
+use super::agent_token::{Decision, Observed, classify_named, is_not_found};
+use super::hive_token::TeamSeen;
+
+/// The builder's forge login.
+pub const LOGIN: &str = "builder";
+
+/// The forge's name for the account's one token.
+pub const TOKEN_NAME: &str = "swarm-builder";
+
+/// What the token may do: read repositories. Which repositories is the
+/// account's business, and the account is only in the `hives` team.
+pub const TOKEN_SCOPES: &str = "read:repository";
+
+/// How often [`spawn`] re-checks the account and its token.
+const RECONCILE_INTERVAL: std::time::Duration = std::time::Duration::from_mins(5);
+
+/// The email this module creates the account with, and the one it requires
+/// before acting on an account named [`LOGIN`]. Under its own domain, so it
+/// equals no agent's or hive's account email.
+const EMAIL: &str = "builder@services.hyperhive.local";
+
+/// Whether a pass that found `seen`, and `member` for the account's place in
+/// the `hives` team, queues a `MintBuilderForgeToken` job.
+/// [`Observed::Unknown`] never does, so a forge or store outage can't rotate
+/// the token, and neither does a [`Decision::Conflict`], which the job would
+/// refuse. A kept token queues only when the account is known to be out of
+/// the team; that job finds the token current and does not rotate it.
+pub fn needs_job(seen: Observed, member: Option<bool>) -> bool {
+    match seen {
+        Observed::NoForgeUser | Observed::Decided(Decision::Mint(_)) => true,
+        Observed::Decided(Decision::Keep) => member == Some(false),
+        Observed::Unknown | Observed::Decided(Decision::Conflict) => false,
+    }
+}
+
+impl Client {
+    /// Make sure [`LOGIN`] exists, is in the `hives` team, and holds a live
+    /// [`TOKEN_NAME`] token in the store, minting one when it does not. The
+    /// whole job of the `MintBuilderForgeToken` node.
+    ///
+    /// # Errors
+    /// When the store or the forge refuses a step, when an account named
+    /// [`LOGIN`] exists with another email, or when the read-back does not
+    /// show the new token.
+    pub async fn ensure_builder_forge_token(&self) -> Result<()> {
+        let path = forge_token_path()?;
+        self.ensure_builder_user().await?;
+        let team = self.ensure_hives_team().await?;
+        self.api
+            .org_add_team_member(team, LOGIN)
+            .await
+            .with_context(|| format!("adding {LOGIN} to the hives team"))?;
+
+        let store = crate::store::connect()
+            .await
+            .context("logging in to the swarm secret store")?;
+        let (stored, listed) = self.read_builder_token(&store, &path).await?;
+        let reason = match classify_named(TOKEN_NAME, TOKEN_SCOPES, stored.as_ref(), &listed) {
+            Decision::Keep => {
+                tracing::debug!("builder forge token is current; left as it is");
+                return Ok(());
+            }
+            Decision::Conflict => bail!(
+                "the forge lists more than one {TOKEN_NAME:?} token for {LOGIN:?}; delete them \
+                 by id and re-run"
+            ),
+            Decision::Mint(reason) => reason,
+        };
+        let credential = self
+            .replace_token(&store, LOGIN, TOKEN_NAME, TOKEN_SCOPES, &listed, &path)
+            .await?;
+
+        Forgejo::new(Auth::Token(&credential.value), self.url.clone())
+            .context("building a forge client with the new token")?
+            .repo_search(RepoSearchQuery::default())
+            .await
+            .with_context(|| format!("using {LOGIN}'s new forge token"))?;
+        let (stored, listed) = self.read_builder_token(&store, &path).await?;
+        let now = classify_named(TOKEN_NAME, TOKEN_SCOPES, stored.as_ref(), &listed);
+        if now != Decision::Keep {
+            bail!("{LOGIN}'s new forge token does not read back as current: {now:?}");
+        }
+        tracing::info!(?reason, %path, "builder forge token minted and stored");
+        Ok(())
+    }
+
+    /// The stored token at `path` and every token the forge lists for
+    /// [`LOGIN`], which must exist.
+    async fn read_builder_token(
+        &self,
+        store: &swarm_secret_client::SecretStore,
+        path: &str,
+    ) -> Result<(
+        Option<forge::Credential>,
+        Vec<forgejo_api::structs::AccessToken>,
+    )> {
+        let stored = store
+            .read_optional(path)
+            .await
+            .with_context(|| format!("reading {path}"))?;
+        let Some(listed) = self.list_agent_tokens(LOGIN).await? else {
+            bail!("the forge has no user {LOGIN:?} after this job created it");
+        };
+        Ok((stored, listed))
+    }
+
+    /// Create [`LOGIN`] if there is none, refuse one with another email, and
+    /// lock it out of creating repos.
+    async fn ensure_builder_user(&self) -> Result<()> {
+        self.create_user(LOGIN, EMAIL).await?;
+        let user = self
+            .api
+            .user_get(LOGIN)
+            .await
+            .with_context(|| format!("read forge user {LOGIN}"))?;
+        if user.email.as_deref() != Some(EMAIL) {
+            bail!(
+                "forge account `{LOGIN}` exists but is not the swarm builder's (email {:?}, \
+                 expected `{EMAIL}`); refusing to give it read on every config repo",
+                user.email
+            );
+        }
+        self.api
+            .admin_edit_user(LOGIN, super::repo_creation_lockdown(LOGIN))
+            .await
+            .with_context(|| format!("disable repo creation for forge user {LOGIN}"))?;
+        Ok(())
+    }
+
+    /// What the store and the forge say about the account's token, and
+    /// whether it is in the `hives` team (`None` when that read failed).
+    async fn observe_builder(&self) -> Result<(Observed, Option<bool>)> {
+        let path = forge_token_path()?;
+        let store = crate::store::connect()
+            .await
+            .context("logging in to the swarm secret store")?;
+        let stored = match store.read_optional::<forge::Credential>(&path).await {
+            Ok(stored) => stored,
+            Err(e) => {
+                tracing::warn!(error = %e, "builder forge token: store read failed");
+                return Ok((Observed::Unknown, None));
+            }
+        };
+        let seen = match self.list_agent_tokens(LOGIN).await {
+            Ok(Some(listed)) => Observed::Decided(classify_named(
+                TOKEN_NAME,
+                TOKEN_SCOPES,
+                stored.as_ref(),
+                &listed,
+            )),
+            Ok(None) => Observed::NoForgeUser,
+            Err(e) => {
+                tracing::warn!(error = %format!("{e:#}"), "builder forge token: forge read failed");
+                Observed::Unknown
+            }
+        };
+        let member = match self.observe_hives_team().await {
+            TeamSeen::Matches(id) | TeamSeen::Drifted(id) => {
+                match self.api.org_list_team_member(id, LOGIN).await {
+                    Ok(_) => Some(true),
+                    Err(e) if is_not_found(&e) => Some(false),
+                    Err(e) => {
+                        tracing::warn!(error = %e, "builder forge token: team membership read failed");
+                        None
+                    }
+                }
+            }
+            TeamSeen::Absent => Some(false),
+            TeamSeen::Unknown => None,
+        };
+        Ok((seen, member))
+    }
+}
+
+/// Check the account now and every [`RECONCILE_INTERVAL`] after, and call
+/// `enqueue`, which inserts a `MintBuilderForgeToken` node, when it needs a
+/// job. A pass that fails is logged and retried on the next tick.
+pub fn spawn(client: Arc<Client>, enqueue: impl Fn() + Send + 'static) {
+    tokio::spawn(async move {
+        let mut ticker = tokio::time::interval(RECONCILE_INTERVAL);
+        loop {
+            ticker.tick().await;
+            let Some(outcome) =
+                super::within_forge_deadline("builder forge token", client.observe_builder()).await
+            else {
+                continue;
+            };
+            match outcome {
+                Ok((seen, member)) if needs_job(seen, member) => {
+                    tracing::info!(?seen, ?member, "builder forge token: queueing a job");
+                    enqueue();
+                }
+                Ok((seen, member)) => {
+                    tracing::debug!(?seen, ?member, "builder forge token: no job");
+                }
+                Err(e) => tracing::warn!(
+                    error = %format!("{e:#}"),
+                    retry_in_s = RECONCILE_INTERVAL.as_secs(),
+                    "builder forge token: pass failed; retrying next tick"
+                ),
+            }
+        }
+    });
+}
+
+#[cfg(test)]
+mod tests {
+    use super::super::agent_token::MintReason;
+    use super::*;
+
+    #[test]
+    fn the_token_reads_repositories_and_nothing_else() {
+        assert_eq!(TOKEN_SCOPES, "read:repository");
+        assert_ne!(TOKEN_NAME, super::super::agent_token::AGENT_TOKEN_NAME);
+        assert_ne!(TOKEN_NAME, super::super::hive_token::HIVE_TOKEN_NAME);
+        assert_ne!(TOKEN_NAME, super::super::classifier_account::TOKEN_NAME);
+    }
+
+    #[test]
+    fn a_mint_a_missing_account_or_a_lost_membership_queues() {
+        let mint = Observed::Decided(Decision::Mint(MintReason::NotStored));
+        let keep = Observed::Decided(Decision::Keep);
+        assert!(needs_job(Observed::NoForgeUser, None));
+        assert!(needs_job(mint, Some(true)));
+        assert!(needs_job(keep, Some(false)));
+        assert!(!needs_job(keep, Some(true)));
+        assert!(!needs_job(keep, None));
+        assert!(!needs_job(
+            Observed::Decided(Decision::Conflict),
+            Some(false)
+        ));
+        assert!(!needs_job(Observed::Unknown, Some(false)));
+    }
+
+    #[test]
+    fn the_accounts_email_is_no_agents_or_hives() {
+        assert_ne!(EMAIL, super::super::agent_email(LOGIN));
+        assert_ne!(EMAIL, super::super::legacy_agent_email(LOGIN));
+        assert!(!LOGIN.starts_with(hive_types::RESERVED_AGENT_PREFIX));
+    }
+
+    /// An existing account named like the builder with an agent's email is
+    /// refused before the lockdown, so it never joins the team.
+    #[tokio::test]
+    async fn the_job_refuses_an_account_that_is_not_the_builders() {
+        use super::super::tests::{lockdown_patch, stub_forge};
+
+        let (client, seen) = stub_forge(
+            (409, r#"{"message":"user already exists"}"#),
+            (200, r#"{"email":"builder@hyperhive.local"}"#),
+            (200, "{}"),
+        )
+        .await;
+        let err = client
+            .ensure_builder_user()
+            .await
+            .expect_err("an agent's account is not the builder's");
+        assert!(format!("{err:#}").contains("not the swarm builder's"));
+        assert!(lockdown_patch(&seen, LOGIN).is_none());
+    }
+}

swarm-controller/src/forge/hive_token.rs

@@ -235,7 +235,7 @@ impl Client {
     /// The id of [`HIVES_TEAM`] in `agent-configs`, creating it or
     /// correcting its settings first when needed. Its members are not
     /// touched here.
-    async fn ensure_hives_team(&self) -> Result<i64> {
+    pub(super) async fn ensure_hives_team(&self) -> Result<i64> {
         match self.observe_hives_team().await {
             TeamSeen::Matches(id) => Ok(id),
             TeamSeen::Drifted(id) => {
@@ -292,7 +292,7 @@ impl Client {
         }
     }
 
-    async fn observe_hives_team(&self) -> TeamSeen {
+    pub(super) async fn observe_hives_team(&self) -> TeamSeen {
         match self.api.org_list_teams(CONFIG_ORG).await {
             Ok((_, teams)) => teams
                 .into_iter()

swarm-controller/src/main.rs

@@ -49,6 +49,7 @@ mod agent_renewal;
 mod agent_state_stream;
 mod agent_status;
 mod auth;
+mod builder;
 mod config_pr;
 mod forge;
 mod forge_account;
@@ -153,6 +154,17 @@ enum SwarmNodeKind {
     /// live token in the swarm store, where this daemon reads it to post the
     /// classifier's reviews. See `forge::classifier_account`.
     MintClassifierForgeToken,
+    /// Make sure the swarm builder's read-only forge account exists, is in
+    /// the `hives` team, and holds a live token in the swarm store, where the
+    /// builder reads it to fetch config repos. See `forge::builder_account`.
+    MintBuilderForgeToken,
+    /// Build `agent`'s config at `rev`, the head of its config PR `pr`, on
+    /// the swarm builder, and write the result as a commit status on `rev`.
+    /// See `builder`.
+    ///
+    /// Carries no hive: the check builds on the builder's own agent base,
+    /// not on any hive.
+    BuildAgent { agent: String, rev: String, pr: u64 },
     /// Re-mint `agent`'s queue secret if it is still stored and old enough.
     /// See `agent_renewal`.
     ///
@@ -194,6 +206,8 @@ impl hive_jobq_wire::WireNode for SwarmNodeKind {
             SwarmNodeKind::MintHiveSenderToken { .. } => "mint_hive_sender_token".to_owned(),
             SwarmNodeKind::MintHiveForgeToken { .. } => "mint_hive_forge_token".to_owned(),
             SwarmNodeKind::MintClassifierForgeToken => "mint_classifier_forge_token".to_owned(),
+            SwarmNodeKind::MintBuilderForgeToken => "mint_builder_forge_token".to_owned(),
+            SwarmNodeKind::BuildAgent { .. } => "build_agent".to_owned(),
             SwarmNodeKind::RenewAgentQueueCredential { .. } => {
                 "renew_agent_queue_credential".to_owned()
             }
@@ -224,7 +238,12 @@ impl hive_jobq_wire::WireNode for SwarmNodeKind {
             }
             SwarmNodeKind::MintHiveSenderToken { hive }
             | SwarmNodeKind::MintHiveForgeToken { hive } => serde_json::json!({ "hive": hive }),
-            SwarmNodeKind::MintClassifierForgeToken => serde_json::json!({}),
+            SwarmNodeKind::MintClassifierForgeToken | SwarmNodeKind::MintBuilderForgeToken => {
+                serde_json::json!({})
+            }
+            SwarmNodeKind::BuildAgent { agent, rev, pr } => {
+                serde_json::json!({ "agent": agent, "rev": rev, "pr": pr })
+            }
             SwarmNodeKind::TriggerDeploy { hive, agent, rev } => {
                 serde_json::json!({ "agent": agent, "hive": hive, "rev": rev })
             }
@@ -382,6 +401,10 @@ async fn run_swarm_node(
         }
         SwarmNodeKind::MintHiveForgeToken { hive } => mint_hive_forge_token(forge, &hive).await,
         SwarmNodeKind::MintClassifierForgeToken => mint_classifier_forge_token(forge).await,
+        SwarmNodeKind::MintBuilderForgeToken => mint_builder_forge_token(forge).await,
+        SwarmNodeKind::BuildAgent { agent, rev, pr } => {
+            build_agent(&deps, forge, agent, rev, pr).await
+        }
         SwarmNodeKind::RenewAgentQueueCredential { agent } => renew_queue_credential(&agent).await,
         SwarmNodeKind::SetAgentWanted { hive, agent } => match deps.wanted {
             None => Outcome::Failed(
@@ -536,6 +559,42 @@ async fn mint_hive_forge_token(
     }
 }
 
+/// The `BuildAgent` arm, lifted out for `clippy::too_many_lines`.
+async fn build_agent(
+    deps: &WorkerDeps,
+    forge: Option<Arc<forge::Client>>,
+    agent: String,
+    rev: String,
+    pr: u64,
+) -> hive_jobq::scheduler::Outcome {
+    let job = swarm_queue_client::builder::BuildAgent { agent, rev, pr };
+    builder::run(deps.queue.as_ref(), forge.as_deref(), job).await
+}
+
+/// The `MintBuilderForgeToken` arm, lifted out for the same reason.
+async fn mint_builder_forge_token(
+    forge: Option<Arc<forge::Client>>,
+) -> hive_jobq::scheduler::Outcome {
+    use hive_jobq::scheduler::Outcome;
+
+    let Some(client) = forge else {
+        return Outcome::Failed(
+            "no forge access here (SWARM_CONTROLLER_FORGE_URL unset, or its token \
+             not in the swarm secret store), so no builder forge token can be minted"
+                .to_owned(),
+        );
+    };
+    match forge_node(
+        "mint_builder_forge_token",
+        client.ensure_builder_forge_token(),
+    )
+    .await
+    {
+        Ok(()) => Outcome::Done,
+        Err(e) => Outcome::Failed(format!("{e:#}")),
+    }
+}
+
 /// The `MintClassifierForgeToken` arm, lifted out for the same reason.
 async fn mint_classifier_forge_token(
     forge: Option<Arc<forge::Client>>,
@@ -1897,6 +1956,33 @@ fn claiming_hives(
         .collect()
 }
 
+/// Queue a build check of a config PR's new head. Every head is built, on
+/// whichever hive the agent runs: the check builds on the builder's own
+/// agent base.
+fn queue_config_build(state: &AppState, changed: config_pr::ChangedConfigPr) {
+    let config_pr::ChangedConfigPr { agent, rev, pr } = changed;
+    let inserted = state
+        .jobq
+        .lock()
+        .unwrap_or_else(std::sync::PoisonError::into_inner)
+        .insert_job(None, |b| {
+            vec![
+                b.node(SwarmNodeKind::BuildAgent {
+                    agent: agent.clone(),
+                    rev: rev.clone(),
+                    pr,
+                })
+                .guid(),
+            ]
+        });
+    match inserted {
+        Ok(_) => tracing::info!(%agent, %rev, pr, "config-pr head: build queued"),
+        Err(e) => {
+            tracing::warn!(%agent, %rev, pr, error = %e, "config-pr head: queueing the build failed");
+        }
+    }
+}
+
 /// Queue the deploy of a merged config PR on the one hive that runs its agent.
 ///
 /// The hive is found by scanning every hive's wanted state, since nothing
@@ -2467,6 +2553,20 @@ fn queue_classifier_forge_token_mint(
         .map_err(|e| anyhow::anyhow!("{e}"))
 }
 
+/// Insert one `MintBuilderForgeToken` job and return its node's id.
+/// `forge::builder_account::spawn`'s periodic pass is the one caller.
+fn queue_builder_forge_token_mint(
+    sched: &Mutex<hive_jobq::scheduler::Scheduler<SwarmNodeKind, SwarmResourceKind>>,
+) -> Result<Vec<hive_jobq::NodeId>> {
+    sched
+        .lock()
+        .unwrap_or_else(std::sync::PoisonError::into_inner)
+        .insert_job(None, |b| {
+            vec![b.node(SwarmNodeKind::MintBuilderForgeToken).guid()]
+        })
+        .map_err(|e| anyhow::anyhow!("{e}"))
+}
+
 /// The homeserver agents' own matrix accounts are minted on. Unset is this
 /// controller creating agents with no matrix account, which the mint node
 /// reports by name rather than `main` refusing to start.
@@ -2509,7 +2609,8 @@ fn spawn_matrix_account_backfill(
 }
 
 /// Start the forge's periodic passes — the swarm-wide objects, the runner
-/// registration token, hives' read tokens, agents' `internal/docs` grants, and
+/// registration token, hives' and the builder's read tokens, agents'
+/// `internal/docs` grants, and
 /// agents' tokens (which also
 /// sweeps legacy tokens; see `forge::legacy_tokens`) —
 /// once there is a forge client. Lifted out of `main` for
@@ -2535,6 +2636,12 @@ fn spawn_forge_workers(
         }
     });
     let sched = Arc::clone(jobq);
+    forge::builder_account::spawn(Arc::clone(&client), move || {
+        if let Err(e) = queue_builder_forge_token_mint(&sched) {
+            tracing::warn!(error = %format!("{e:#}"), "builder forge token: queueing failed");
+        }
+    });
+    let sched = Arc::clone(jobq);
     forge::docs_grant::spawn(Arc::clone(&client), move |agents| {
         if let Err(e) = queue_docs_grants(&sched, agents) {
             tracing::warn!(error = %format!("{e:#}"), "docs grants: queueing failed");
@@ -4262,6 +4369,10 @@ mod tests {
                 "mint_classifier_forge_token",
                 SwarmNodeKind::MintClassifierForgeToken,
             ),
+            (
+                "mint_builder_forge_token",
+                SwarmNodeKind::MintBuilderForgeToken,
+            ),
         ];
         for (label, kind) in nodes {
             let mut sched = hive_jobq::scheduler::Scheduler::new(

swarm-controller/src/webhook.rs

@@ -418,6 +418,9 @@ pub(super) async fn post_webhook_forge(
             if let Some(merged) = crate::config_pr::merged(&body) {
                 super::queue_merged_config_deploy(&state, merged).await;
             }
+            if let Some(changed) = crate::config_pr::changed_head(&body) {
+                super::queue_config_build(&state, changed);
+            }
         }
         DeliveryKind::VcsActivity => {
             if let Some(activity) = crate::vcs_metrics::parse_push(&body) {