swarm-builder: build each agent config PR head and post a commit status

A new `swarm-builder` service, in its own container on the host that sets
`deploy.builder.enable` (defaulted on by `allSwarmServices` where the host
has the builder's store leaf, like the review classifier).

When an `agent-configs/<agent>` PR is opened, reopened or synchronized,
swarm-controller queues a `build_agent` jobq node. The node sets a
`swarm-builder` commit status to pending on the PR head, sends
`{agent, rev, pr}` to the builder over NATS request-reply
(`$SWARM.builder.build`), and writes success (store path), failure (last
line nix printed) or error (no builder / no reply) on the head. The node
fails when the build does, so the Jobs page matches the PR.

The builder builds nix/builder/check-flake's `check` with every input
overridden: hyperhive and nixpkgs to the host's own
`c0re.hyperhiveFlake`/`c0re.nixpkgsFlake` (what a hive there renders into
its meta flake), and the agent to the PR head on the forge. Evaluation is
pure and runs in the container; builds go to the host nix daemon as an
untrusted client. The private config repo is fetched with the token of a
new read-only `builder` forge account (in the `hives` team), which the
controller mints into `swarm/services/builder/forge/token`; git gets it
from a credential helper reading the child's environment, so it is in no
argument or URL.

Credentials: a `builder` store leaf minted by glue-bao-tls on the store
host, a `swarm-builder` reader policy (forge token + its own OIDC client
secret), an authelia machine client, and a `--builder-client` grant in the
queue's auth-callout responder. The controller gains a grant on the token
path and the secret publisher a deny on it.

This is the agent-config build check atlas scoped on #5081 (slice 3):
nothing is pushed to a cache, nothing is written to KV, and a red status
gates no deploy. The check composes agent-base + the config without the
values a hive adds (CA, docs, claude-code path, ports, input follows), so
it can disagree with a hive's own build.

swarm-controller/src/builder.rs

@@ -0,0 +1,304 @@
+//! The `BuildAgent` node: send a config PR's head to the swarm builder over the
+//! queue, and turn its answer into the [`CONTEXT`] commit status on that head.
+//!
+//! The status is the check's only record besides the node itself: nothing is
+//! pushed, and the closure the builder made stays in its host's store. A red
+//! status blocks nothing; the merge deploys as before.
+
+use std::future::Future;
+
+use forgejo_api::structs::CommitStatusState;
+use hive_jobq::scheduler::Outcome;
+use swarm_queue_client::builder::{BuildAgent, REPLY_DEADLINE, Reply, SUBJECT};
+
+use crate::forge::{self, CONFIG_ORG};
+
+/// The commit status context the check writes.
+pub const CONTEXT: &str = "swarm-builder";
+
+/// Where a build is sent: the queue in production, a fake in the tests.
+/// `Err` is a build that never reached a builder or never came back.
+pub(crate) trait BuildSink {
+    fn build(&self, request: Vec<u8>) -> impl Future<Output = Result<Reply, String>> + Send;
+}
+
+impl BuildSink for async_nats::Client {
+    async fn build(&self, request: Vec<u8>) -> Result<Reply, String> {
+        use async_nats::RequestErrorKind;
+
+        let sent = async_nats::Request::new()
+            .payload(request.into())
+            .timeout(Some(REPLY_DEADLINE));
+        match self.send_request(SUBJECT, sent).await {
+            Ok(msg) => serde_json::from_slice(&msg.payload)
+                .map_err(|e| format!("the builder's reply did not decode: {e}")),
+            Err(e) => Err(match e.kind() {
+                RequestErrorKind::NoResponders => "no swarm builder is listening: \
+                     deploy.builder.enable is set on no host, or its service is not connected \
+                     to the swarm queue"
+                    .to_owned(),
+                RequestErrorKind::TimedOut => format!(
+                    "the swarm builder sent no answer within {} minutes",
+                    REPLY_DEADLINE.as_secs() / 60
+                ),
+                _ => format!("sending the build to the swarm builder failed: {e}"),
+            }),
+        }
+    }
+}
+
+/// Where a status is written: the forge in production, a fake in the tests.
+pub(crate) trait StatusSink {
+    fn set(
+        &self,
+        agent: &str,
+        rev: &str,
+        state: CommitStatusState,
+        description: &str,
+    ) -> impl Future<Output = anyhow::Result<()>> + Send;
+}
+
+impl StatusSink for forge::Client {
+    async fn set(
+        &self,
+        agent: &str,
+        rev: &str,
+        state: CommitStatusState,
+        description: &str,
+    ) -> anyhow::Result<()> {
+        forge::within_forge_deadline(
+            "build_agent status",
+            self.create_commit_status(CONFIG_ORG, agent, rev, CONTEXT, state, description),
+        )
+        .await
+        .unwrap_or_else(|| Err(anyhow::anyhow!("no answer from the forge")))
+    }
+}
+
+/// Build `job` and write its status. The node fails when the build does, so
+/// the Jobs page shows red where the PR does.
+pub(crate) async fn run(
+    queue: Option<&impl BuildSink>,
+    statuses: Option<&impl StatusSink>,
+    job: BuildAgent,
+) -> Outcome {
+    let Some(statuses) = statuses else {
+        return Outcome::Failed(
+            "no forge access here (SWARM_CONTROLLER_FORGE_URL unset, or its token not in the \
+             swarm secret store), so no build status can be written"
+                .to_owned(),
+        );
+    };
+    let set = |state, description: String| {
+        let job = &job;
+        async move {
+            statuses
+                .set(&job.agent, &job.rev, state, &description)
+                .await
+                .map_err(|e| format!("{e:#}"))
+        }
+    };
+    let Some(queue) = queue else {
+        let error = "no swarm queue is configured on this host, so no build can be sent";
+        return settle(set(CommitStatusState::Error, error.to_owned()).await, error);
+    };
+    if let Err(e) = set(
+        CommitStatusState::Pending,
+        "building on the swarm builder".to_owned(),
+    )
+    .await
+    {
+        tracing::warn!(agent = %job.agent, rev = %job.rev, error = %e, "build_agent: the pending status was not written");
+    }
+    let request = match serde_json::to_vec(&job) {
+        Ok(request) => request,
+        Err(e) => return Outcome::Failed(format!("encoding the build: {e}")),
+    };
+    match queue.build(request).await {
+        Ok(Reply::Built { out_path }) => {
+            tracing::info!(agent = %job.agent, rev = %job.rev, %out_path, "build_agent: built");
+            match set(CommitStatusState::Success, out_path).await {
+                Ok(()) => Outcome::Done,
+                Err(e) => Outcome::Failed(format!("built, but the status was not written: {e}")),
+            }
+        }
+        Ok(Reply::Failed { error_tail }) => {
+            let last = last_line(&error_tail);
+            settle(
+                set(CommitStatusState::Failure, last.to_owned()).await,
+                &format!("the build failed: {error_tail}"),
+            )
+        }
+        Err(error) => settle(set(CommitStatusState::Error, error.clone()).await, &error),
+    }
+}
+
+/// A failed node carrying `error`, and the status write's own failure when
+/// there was one.
+fn settle(written: Result<(), String>, error: &str) -> Outcome {
+    match written {
+        Ok(()) => Outcome::Failed(error.to_owned()),
+        Err(e) => Outcome::Failed(format!("{error} (and the status was not written: {e})")),
+    }
+}
+
+/// The last non-blank line of `tail`, which for `nix` is the error itself.
+fn last_line(tail: &str) -> &str {
+    tail.lines()
+        .rev()
+        .map(str::trim)
+        .find(|l| !l.is_empty())
+        .unwrap_or("the build failed")
+}
+
+#[cfg(test)]
+mod tests {
+    use std::sync::Mutex;
+
+    use super::*;
+
+    struct FakeQueue(Result<Reply, String>);
+
+    impl BuildSink for FakeQueue {
+        async fn build(&self, request: Vec<u8>) -> Result<Reply, String> {
+            let sent: BuildAgent = serde_json::from_slice(&request).expect("a build request");
+            assert_eq!(sent, job());
+            self.0.clone()
+        }
+    }
+
+    #[derive(Default)]
+    struct FakeStatuses(Mutex<Vec<(CommitStatusState, String)>>);
+
+    impl StatusSink for FakeStatuses {
+        async fn set(
+            &self,
+            agent: &str,
+            rev: &str,
+            state: CommitStatusState,
+            description: &str,
+        ) -> anyhow::Result<()> {
+            assert_eq!((agent, rev), ("atlas", "abc"));
+            self.0.lock().unwrap().push((state, description.to_owned()));
+            Ok(())
+        }
+    }
+
+    fn job() -> BuildAgent {
+        BuildAgent {
+            agent: "atlas".to_owned(),
+            rev: "abc".to_owned(),
+            pr: 3,
+        }
+    }
+
+    async fn statuses_after(queue: Option<FakeQueue>) -> (Outcome, Vec<CommitStatusState>) {
+        let statuses = FakeStatuses::default();
+        let outcome = run(queue.as_ref(), Some(&statuses), job()).await;
+        let states = statuses.0.into_inner().unwrap();
+        (outcome, states.into_iter().map(|(s, _)| s).collect())
+    }
+
+    #[tokio::test]
+    async fn a_built_config_goes_pending_then_green() {
+        let (outcome, states) = statuses_after(Some(FakeQueue(Ok(Reply::Built {
+            out_path: "/nix/store/x".to_owned(),
+        }))))
+        .await;
+        assert!(matches!(outcome, Outcome::Done));
+        assert_eq!(
+            states,
+            [CommitStatusState::Pending, CommitStatusState::Success]
+        );
+    }
+
+    #[tokio::test]
+    async fn a_failed_build_goes_red_with_its_last_line_and_fails_the_node() {
+        let statuses = FakeStatuses::default();
+        let queue = FakeQueue(Ok(Reply::Failed {
+            error_tail: "evaluating\nerror: boom\n".to_owned(),
+        }));
+        let outcome = run(Some(&queue), Some(&statuses), job()).await;
+        assert!(matches!(&outcome, Outcome::Failed(e) if e.contains("error: boom")));
+        let written = statuses.0.into_inner().unwrap();
+        assert_eq!(
+            written.last(),
+            Some(&(CommitStatusState::Failure, "error: boom".to_owned()))
+        );
+    }
+
+    #[tokio::test]
+    async fn no_builder_is_an_error_status_not_a_failure() {
+        let (outcome, states) =
+            statuses_after(Some(FakeQueue(Err("no swarm builder".to_owned())))).await;
+        assert!(matches!(outcome, Outcome::Failed(_)));
+        assert_eq!(
+            states,
+            [CommitStatusState::Pending, CommitStatusState::Error]
+        );
+    }
+
+    #[tokio::test]
+    async fn no_queue_is_an_error_status() {
+        let (outcome, states) = statuses_after(None).await;
+        assert!(matches!(&outcome, Outcome::Failed(e) if e.contains("no swarm queue")));
+        assert_eq!(states, [CommitStatusState::Error]);
+    }
+
+    #[tokio::test]
+    async fn no_forge_fails_the_node_without_sending_the_build() {
+        struct Unreachable;
+        impl BuildSink for Unreachable {
+            async fn build(&self, _: Vec<u8>) -> Result<Reply, String> {
+                panic!("a build with nowhere to report must not be sent")
+            }
+        }
+        let outcome = run(Some(&Unreachable), None::<&FakeStatuses>, job()).await;
+        assert!(matches!(&outcome, Outcome::Failed(e) if e.contains("no forge access")));
+    }
+
+    /// The queue half against a real server: no builder listening names the
+    /// option, and a listener's reply comes back decoded.
+    #[tokio::test]
+    async fn a_build_reaches_a_listening_builder_and_fails_without_one() {
+        use futures_util::StreamExt as _;
+
+        let nats = crate::test_nats::TestNats::start();
+        let client = nats.client().await;
+        let unanswered = client.build(b"{}".to_vec()).await;
+        assert!(
+            matches!(&unanswered, Err(e) if e.contains("deploy.builder.enable")),
+            "{unanswered:?}"
+        );
+
+        let builder = nats.client().await;
+        let mut requests = builder
+            .queue_subscribe(SUBJECT, swarm_queue_client::builder::QUEUE_GROUP.to_owned())
+            .await
+            .expect("subscribe");
+        // A message the builder sends itself and receives back proves the
+        // server processed the subscription before it.
+        let mut probe = builder
+            .subscribe("builder-test.probe")
+            .await
+            .expect("subscribe");
+        builder
+            .publish("builder-test.probe", "".into())
+            .await
+            .expect("publish");
+        probe.next().await.expect("the probe comes back");
+        let built = Reply::Built {
+            out_path: "/nix/store/x".to_owned(),
+        };
+        let answer = built.clone();
+        tokio::spawn(async move {
+            let msg = requests.next().await.expect("a request");
+            let reply = serde_json::to_vec(&answer).expect("encodes");
+            builder
+                .publish(msg.reply.expect("a reply subject"), reply.into())
+                .await
+                .expect("publish");
+        });
+        assert_eq!(client.build(b"{}".to_vec()).await, Ok(built));
+    }
+}