swarm-builder: build each agent config PR head and post a commit status

A new `swarm-builder` service, in its own container on the host that sets
`deploy.builder.enable` (defaulted on by `allSwarmServices` where the host
has the builder's store leaf, like the review classifier).

When an `agent-configs/<agent>` PR is opened, reopened or synchronized,
swarm-controller queues a `build_agent` jobq node. The node sets a
`swarm-builder` commit status to pending on the PR head, sends
`{agent, rev, pr}` to the builder over NATS request-reply
(`$SWARM.builder.build`), and writes success (store path), failure (last
line nix printed) or error (no builder / no reply) on the head. The node
fails when the build does, so the Jobs page matches the PR.

The builder builds nix/builder/check-flake's `check` with every input
overridden: hyperhive and nixpkgs to the host's own
`c0re.hyperhiveFlake`/`c0re.nixpkgsFlake` (what a hive there renders into
its meta flake), and the agent to the PR head on the forge. Evaluation is
pure and runs in the container; builds go to the host nix daemon as an
untrusted client. The private config repo is fetched with the token of a
new read-only `builder` forge account (in the `hives` team), which the
controller mints into `swarm/services/builder/forge/token`; git gets it
from a credential helper reading the child's environment, so it is in no
argument or URL.

Credentials: a `builder` store leaf minted by glue-bao-tls on the store
host, a `swarm-builder` reader policy (forge token + its own OIDC client
secret), an authelia machine client, and a `--builder-client` grant in the
queue's auth-callout responder. The controller gains a grant on the token
path and the secret publisher a deny on it.

This is the agent-config build check atlas scoped on #5081 (slice 3):
nothing is pushed to a cache, nothing is written to KV, and a red status
gates no deploy. The check composes agent-base + the config without the
values a hive adds (CA, docs, claude-code path, ports, input follows), so
it can disagree with a hive's own build.

swarm-controller/src/forge/builder_account.rs

@@ -0,0 +1,284 @@
+//! The swarm builder's read-only forge account, [`LOGIN`]: created here with
+//! the admin API, put in the `hives` team of `agent-configs` (read on every
+//! config repo, including ones created later), and given one token, stored
+//! at `swarm_secret_client::builder::forge_token_path`, where the builder
+//! reads it under its own store identity to fetch the config it builds.
+//!
+//! The builder never writes to the forge: its build results become commit
+//! statuses through this daemon's own client (`crate::builder`).
+//!
+//! The token decision is [`super::agent_token`]'s, under this module's name
+//! and scopes, so a stored token the forge still lists with the same last
+//! eight characters and scopes is kept, and a pass whose reads fail queues
+//! nothing.
+
+use std::sync::Arc;
+
+use anyhow::{Context, Result, bail};
+use forgejo_api::structs::RepoSearchQuery;
+use forgejo_api::{Auth, Forgejo};
+use swarm_secret_client::builder::forge_token_path;
+use swarm_secret_client::forge;
+
+use super::Client;
+use super::agent_token::{Decision, Observed, classify_named, is_not_found};
+use super::hive_token::TeamSeen;
+
+/// The builder's forge login.
+pub const LOGIN: &str = "builder";
+
+/// The forge's name for the account's one token.
+pub const TOKEN_NAME: &str = "swarm-builder";
+
+/// What the token may do: read repositories. Which repositories is the
+/// account's business, and the account is only in the `hives` team.
+pub const TOKEN_SCOPES: &str = "read:repository";
+
+/// How often [`spawn`] re-checks the account and its token.
+const RECONCILE_INTERVAL: std::time::Duration = std::time::Duration::from_mins(5);
+
+/// The email this module creates the account with, and the one it requires
+/// before acting on an account named [`LOGIN`]. Under its own domain, so it
+/// equals no agent's or hive's account email.
+const EMAIL: &str = "builder@services.hyperhive.local";
+
+/// Whether a pass that found `seen`, and `member` for the account's place in
+/// the `hives` team, queues a `MintBuilderForgeToken` job.
+/// [`Observed::Unknown`] never does, so a forge or store outage can't rotate
+/// the token, and neither does a [`Decision::Conflict`], which the job would
+/// refuse. A kept token queues only when the account is known to be out of
+/// the team; that job finds the token current and does not rotate it.
+pub fn needs_job(seen: Observed, member: Option<bool>) -> bool {
+    match seen {
+        Observed::NoForgeUser | Observed::Decided(Decision::Mint(_)) => true,
+        Observed::Decided(Decision::Keep) => member == Some(false),
+        Observed::Unknown | Observed::Decided(Decision::Conflict) => false,
+    }
+}
+
+impl Client {
+    /// Make sure [`LOGIN`] exists, is in the `hives` team, and holds a live
+    /// [`TOKEN_NAME`] token in the store, minting one when it does not. The
+    /// whole job of the `MintBuilderForgeToken` node.
+    ///
+    /// # Errors
+    /// When the store or the forge refuses a step, when an account named
+    /// [`LOGIN`] exists with another email, or when the read-back does not
+    /// show the new token.
+    pub async fn ensure_builder_forge_token(&self) -> Result<()> {
+        let path = forge_token_path()?;
+        self.ensure_builder_user().await?;
+        let team = self.ensure_hives_team().await?;
+        self.api
+            .org_add_team_member(team, LOGIN)
+            .await
+            .with_context(|| format!("adding {LOGIN} to the hives team"))?;
+
+        let store = crate::store::connect()
+            .await
+            .context("logging in to the swarm secret store")?;
+        let (stored, listed) = self.read_builder_token(&store, &path).await?;
+        let reason = match classify_named(TOKEN_NAME, TOKEN_SCOPES, stored.as_ref(), &listed) {
+            Decision::Keep => {
+                tracing::debug!("builder forge token is current; left as it is");
+                return Ok(());
+            }
+            Decision::Conflict => bail!(
+                "the forge lists more than one {TOKEN_NAME:?} token for {LOGIN:?}; delete them \
+                 by id and re-run"
+            ),
+            Decision::Mint(reason) => reason,
+        };
+        let credential = self
+            .replace_token(&store, LOGIN, TOKEN_NAME, TOKEN_SCOPES, &listed, &path)
+            .await?;
+
+        Forgejo::new(Auth::Token(&credential.value), self.url.clone())
+            .context("building a forge client with the new token")?
+            .repo_search(RepoSearchQuery::default())
+            .await
+            .with_context(|| format!("using {LOGIN}'s new forge token"))?;
+        let (stored, listed) = self.read_builder_token(&store, &path).await?;
+        let now = classify_named(TOKEN_NAME, TOKEN_SCOPES, stored.as_ref(), &listed);
+        if now != Decision::Keep {
+            bail!("{LOGIN}'s new forge token does not read back as current: {now:?}");
+        }
+        tracing::info!(?reason, %path, "builder forge token minted and stored");
+        Ok(())
+    }
+
+    /// The stored token at `path` and every token the forge lists for
+    /// [`LOGIN`], which must exist.
+    async fn read_builder_token(
+        &self,
+        store: &swarm_secret_client::SecretStore,
+        path: &str,
+    ) -> Result<(
+        Option<forge::Credential>,
+        Vec<forgejo_api::structs::AccessToken>,
+    )> {
+        let stored = store
+            .read_optional(path)
+            .await
+            .with_context(|| format!("reading {path}"))?;
+        let Some(listed) = self.list_agent_tokens(LOGIN).await? else {
+            bail!("the forge has no user {LOGIN:?} after this job created it");
+        };
+        Ok((stored, listed))
+    }
+
+    /// Create [`LOGIN`] if there is none, refuse one with another email, and
+    /// lock it out of creating repos.
+    async fn ensure_builder_user(&self) -> Result<()> {
+        self.create_user(LOGIN, EMAIL).await?;
+        let user = self
+            .api
+            .user_get(LOGIN)
+            .await
+            .with_context(|| format!("read forge user {LOGIN}"))?;
+        if user.email.as_deref() != Some(EMAIL) {
+            bail!(
+                "forge account `{LOGIN}` exists but is not the swarm builder's (email {:?}, \
+                 expected `{EMAIL}`); refusing to give it read on every config repo",
+                user.email
+            );
+        }
+        self.api
+            .admin_edit_user(LOGIN, super::repo_creation_lockdown(LOGIN))
+            .await
+            .with_context(|| format!("disable repo creation for forge user {LOGIN}"))?;
+        Ok(())
+    }
+
+    /// What the store and the forge say about the account's token, and
+    /// whether it is in the `hives` team (`None` when that read failed).
+    async fn observe_builder(&self) -> Result<(Observed, Option<bool>)> {
+        let path = forge_token_path()?;
+        let store = crate::store::connect()
+            .await
+            .context("logging in to the swarm secret store")?;
+        let stored = match store.read_optional::<forge::Credential>(&path).await {
+            Ok(stored) => stored,
+            Err(e) => {
+                tracing::warn!(error = %e, "builder forge token: store read failed");
+                return Ok((Observed::Unknown, None));
+            }
+        };
+        let seen = match self.list_agent_tokens(LOGIN).await {
+            Ok(Some(listed)) => Observed::Decided(classify_named(
+                TOKEN_NAME,
+                TOKEN_SCOPES,
+                stored.as_ref(),
+                &listed,
+            )),
+            Ok(None) => Observed::NoForgeUser,
+            Err(e) => {
+                tracing::warn!(error = %format!("{e:#}"), "builder forge token: forge read failed");
+                Observed::Unknown
+            }
+        };
+        let member = match self.observe_hives_team().await {
+            TeamSeen::Matches(id) | TeamSeen::Drifted(id) => {
+                match self.api.org_list_team_member(id, LOGIN).await {
+                    Ok(_) => Some(true),
+                    Err(e) if is_not_found(&e) => Some(false),
+                    Err(e) => {
+                        tracing::warn!(error = %e, "builder forge token: team membership read failed");
+                        None
+                    }
+                }
+            }
+            TeamSeen::Absent => Some(false),
+            TeamSeen::Unknown => None,
+        };
+        Ok((seen, member))
+    }
+}
+
+/// Check the account now and every [`RECONCILE_INTERVAL`] after, and call
+/// `enqueue`, which inserts a `MintBuilderForgeToken` node, when it needs a
+/// job. A pass that fails is logged and retried on the next tick.
+pub fn spawn(client: Arc<Client>, enqueue: impl Fn() + Send + 'static) {
+    tokio::spawn(async move {
+        let mut ticker = tokio::time::interval(RECONCILE_INTERVAL);
+        loop {
+            ticker.tick().await;
+            let Some(outcome) =
+                super::within_forge_deadline("builder forge token", client.observe_builder()).await
+            else {
+                continue;
+            };
+            match outcome {
+                Ok((seen, member)) if needs_job(seen, member) => {
+                    tracing::info!(?seen, ?member, "builder forge token: queueing a job");
+                    enqueue();
+                }
+                Ok((seen, member)) => {
+                    tracing::debug!(?seen, ?member, "builder forge token: no job");
+                }
+                Err(e) => tracing::warn!(
+                    error = %format!("{e:#}"),
+                    retry_in_s = RECONCILE_INTERVAL.as_secs(),
+                    "builder forge token: pass failed; retrying next tick"
+                ),
+            }
+        }
+    });
+}
+
+#[cfg(test)]
+mod tests {
+    use super::super::agent_token::MintReason;
+    use super::*;
+
+    #[test]
+    fn the_token_reads_repositories_and_nothing_else() {
+        assert_eq!(TOKEN_SCOPES, "read:repository");
+        assert_ne!(TOKEN_NAME, super::super::agent_token::AGENT_TOKEN_NAME);
+        assert_ne!(TOKEN_NAME, super::super::hive_token::HIVE_TOKEN_NAME);
+        assert_ne!(TOKEN_NAME, super::super::classifier_account::TOKEN_NAME);
+    }
+
+    #[test]
+    fn a_mint_a_missing_account_or_a_lost_membership_queues() {
+        let mint = Observed::Decided(Decision::Mint(MintReason::NotStored));
+        let keep = Observed::Decided(Decision::Keep);
+        assert!(needs_job(Observed::NoForgeUser, None));
+        assert!(needs_job(mint, Some(true)));
+        assert!(needs_job(keep, Some(false)));
+        assert!(!needs_job(keep, Some(true)));
+        assert!(!needs_job(keep, None));
+        assert!(!needs_job(
+            Observed::Decided(Decision::Conflict),
+            Some(false)
+        ));
+        assert!(!needs_job(Observed::Unknown, Some(false)));
+    }
+
+    #[test]
+    fn the_accounts_email_is_no_agents_or_hives() {
+        assert_ne!(EMAIL, super::super::agent_email(LOGIN));
+        assert_ne!(EMAIL, super::super::legacy_agent_email(LOGIN));
+        assert!(!LOGIN.starts_with(hive_types::RESERVED_AGENT_PREFIX));
+    }
+
+    /// An existing account named like the builder with an agent's email is
+    /// refused before the lockdown, so it never joins the team.
+    #[tokio::test]
+    async fn the_job_refuses_an_account_that_is_not_the_builders() {
+        use super::super::tests::{lockdown_patch, stub_forge};
+
+        let (client, seen) = stub_forge(
+            (409, r#"{"message":"user already exists"}"#),
+            (200, r#"{"email":"builder@hyperhive.local"}"#),
+            (200, "{}"),
+        )
+        .await;
+        let err = client
+            .ensure_builder_user()
+            .await
+            .expect_err("an agent's account is not the builder's");
+        assert!(format!("{err:#}").contains("not the swarm builder's"));
+        assert!(lockdown_patch(&seen, LOGIN).is_none());
+    }
+}

swarm-controller/src/forge/hive_token.rs

@@ -235,7 +235,7 @@ impl Client {
     /// The id of [`HIVES_TEAM`] in `agent-configs`, creating it or
     /// correcting its settings first when needed. Its members are not
     /// touched here.
-    async fn ensure_hives_team(&self) -> Result<i64> {
+    pub(super) async fn ensure_hives_team(&self) -> Result<i64> {
         match self.observe_hives_team().await {
             TeamSeen::Matches(id) => Ok(id),
             TeamSeen::Drifted(id) => {
@@ -292,7 +292,7 @@ impl Client {
         }
     }
 
-    async fn observe_hives_team(&self) -> TeamSeen {
+    pub(super) async fn observe_hives_team(&self) -> TeamSeen {
         match self.api.org_list_teams(CONFIG_ORG).await {
             Ok((_, teams)) => teams
                 .into_iter()