vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-builder: build each agent config PR head and post a commit status

Browse source 
A new `swarm-builder` service, in its own container on the host that sets
`deploy.builder.enable` (defaulted on by `allSwarmServices` where the host
has the builder's store leaf, like the review classifier).

When an `agent-configs/<agent>` PR is opened, reopened or synchronized,
swarm-controller queues a `build_agent` jobq node. The node sets a
`swarm-builder` commit status to pending on the PR head, sends
`{agent, rev, pr}` to the builder over NATS request-reply
(`$SWARM.builder.build`), and writes success (store path), failure (last
line nix printed) or error (no builder / no reply) on the head. The node
fails when the build does, so the Jobs page matches the PR.

The builder builds nix/builder/check-flake's `check` with every input
overridden: hyperhive and nixpkgs to the host's own
`c0re.hyperhiveFlake`/`c0re.nixpkgsFlake` (what a hive there renders into
its meta flake), and the agent to the PR head on the forge. Evaluation is
pure and runs in the container; builds go to the host nix daemon as an
untrusted client. The private config repo is fetched with the token of a
new read-only `builder` forge account (in the `hives` team), which the
controller mints into `swarm/services/builder/forge/token`; git gets it
from a credential helper reading the child's environment, so it is in no
argument or URL.

Credentials: a `builder` store leaf minted by glue-bao-tls on the store
host, a `swarm-builder` reader policy (forge token + its own OIDC client
secret), an authelia machine client, and a `--builder-client` grant in the
queue's auth-callout responder. The controller gains a grant on the token
path and the secret publisher a deny on it.

This is the agent-config build check atlas scoped on #5081 (slice 3):
nothing is pushed to a cache, nothing is written to KV, and a red status
gates no deploy. The check composes agent-base + the config without the
values a hive adds (CA, docs, claude-code path, ports, input follows), so
it can disagree with a hive's own build.
This commit is contained in:
atlas 2026-10-12 22:04:51 +02:00
commit b68b74d687
43 changed files with 2452 additions and 42 deletions
Download patch file Download diff file Expand all files Collapse all files

115
swarm-controller/src/main.rs
View file

@ -49,6 +49,7 @@ mod agent_renewal;
mod agent_state_stream;
mod agent_status;
mod auth;
mod builder;
mod config_pr;
mod forge;
mod forge_account;
@ -153,6 +154,17 @@ enum SwarmNodeKind {
/// live token in the swarm store, where this daemon reads it to post the
/// classifier's reviews. See `forge::classifier_account`.
MintClassifierForgeToken,
/// Make sure the swarm builder's read-only forge account exists, is in
/// the `hives` team, and holds a live token in the swarm store, where the
/// builder reads it to fetch config repos. See `forge::builder_account`.
MintBuilderForgeToken,
/// Build `agent`'s config at `rev`, the head of its config PR `pr`, on
/// the swarm builder, and write the result as a commit status on `rev`.
/// See `builder`.
///
/// Carries no hive: the check builds on the builder's own agent base,
/// not on any hive.
BuildAgent { agent: String, rev: String, pr: u64 },
/// Re-mint `agent`'s queue secret if it is still stored and old enough.
/// See `agent_renewal`.
///
@ -194,6 +206,8 @@ impl hive_jobq_wire::WireNode for SwarmNodeKind {
SwarmNodeKind::MintHiveSenderToken { .. } => "mint_hive_sender_token".to_owned(),
SwarmNodeKind::MintHiveForgeToken { .. } => "mint_hive_forge_token".to_owned(),
SwarmNodeKind::MintClassifierForgeToken => "mint_classifier_forge_token".to_owned(),
SwarmNodeKind::MintBuilderForgeToken => "mint_builder_forge_token".to_owned(),
SwarmNodeKind::BuildAgent { .. } => "build_agent".to_owned(),
SwarmNodeKind::RenewAgentQueueCredential { .. } => {
"renew_agent_queue_credential".to_owned()
}
@ -224,7 +238,12 @@ impl hive_jobq_wire::WireNode for SwarmNodeKind {
}
SwarmNodeKind::MintHiveSenderToken { hive }
| SwarmNodeKind::MintHiveForgeToken { hive } => serde_json::json!({ "hive": hive }),
SwarmNodeKind::MintClassifierForgeToken => serde_json::json!({}),
SwarmNodeKind::MintClassifierForgeToken | SwarmNodeKind::MintBuilderForgeToken => {
serde_json::json!({})
}
SwarmNodeKind::BuildAgent { agent, rev, pr } => {
serde_json::json!({ "agent": agent, "rev": rev, "pr": pr })
}
SwarmNodeKind::TriggerDeploy { hive, agent, rev } => {
serde_json::json!({ "agent": agent, "hive": hive, "rev": rev })
}
@ -382,6 +401,10 @@ async fn run_swarm_node(
}
SwarmNodeKind::MintHiveForgeToken { hive } => mint_hive_forge_token(forge, &hive).await,
SwarmNodeKind::MintClassifierForgeToken => mint_classifier_forge_token(forge).await,
SwarmNodeKind::MintBuilderForgeToken => mint_builder_forge_token(forge).await,
SwarmNodeKind::BuildAgent { agent, rev, pr } => {
build_agent(&deps, forge, agent, rev, pr).await
}
SwarmNodeKind::RenewAgentQueueCredential { agent } => renew_queue_credential(&agent).await,
SwarmNodeKind::SetAgentWanted { hive, agent } => match deps.wanted {
None => Outcome::Failed(
@ -536,6 +559,42 @@ async fn mint_hive_forge_token(
}
}
/// The `BuildAgent` arm, lifted out for `clippy::too_many_lines`.
async fn build_agent(
deps: &WorkerDeps,
forge: Option<Arc<forge::Client>>,
agent: String,
rev: String,
pr: u64,
) -> hive_jobq::scheduler::Outcome {
let job = swarm_queue_client::builder::BuildAgent { agent, rev, pr };
builder::run(deps.queue.as_ref(), forge.as_deref(), job).await
}
/// The `MintBuilderForgeToken` arm, lifted out for the same reason.
async fn mint_builder_forge_token(
forge: Option<Arc<forge::Client>>,
) -> hive_jobq::scheduler::Outcome {
use hive_jobq::scheduler::Outcome;
let Some(client) = forge else {
return Outcome::Failed(
"no forge access here (SWARM_CONTROLLER_FORGE_URL unset, or its token \
not in the swarm secret store), so no builder forge token can be minted"
.to_owned(),
);
};
match forge_node(
"mint_builder_forge_token",
client.ensure_builder_forge_token(),
)
.await
{
Ok(()) => Outcome::Done,
Err(e) => Outcome::Failed(format!("{e:#}")),
}
}
/// The `MintClassifierForgeToken` arm, lifted out for the same reason.
async fn mint_classifier_forge_token(
forge: Option<Arc<forge::Client>>,
@ -1897,6 +1956,33 @@ fn claiming_hives(
.collect()
}
/// Queue a build check of a config PR's new head. Every head is built, on
/// whichever hive the agent runs: the check builds on the builder's own
/// agent base.
fn queue_config_build(state: &AppState, changed: config_pr::ChangedConfigPr) {
let config_pr::ChangedConfigPr { agent, rev, pr } = changed;
let inserted = state
.jobq
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.insert_job(None, |b| {
vec![
b.node(SwarmNodeKind::BuildAgent {
agent: agent.clone(),
rev: rev.clone(),
pr,
})
.guid(),
]
});
match inserted {
Ok(_) => tracing::info!(%agent, %rev, pr, "config-pr head: build queued"),
Err(e) => {
tracing::warn!(%agent, %rev, pr, error = %e, "config-pr head: queueing the build failed");
}
}
}
/// Queue the deploy of a merged config PR on the one hive that runs its agent.
///
/// The hive is found by scanning every hive's wanted state, since nothing
@ -2467,6 +2553,20 @@ fn queue_classifier_forge_token_mint(
.map_err(|e| anyhow::anyhow!("{e}"))
}
/// Insert one `MintBuilderForgeToken` job and return its node's id.
/// `forge::builder_account::spawn`'s periodic pass is the one caller.
fn queue_builder_forge_token_mint(
sched: &Mutex<hive_jobq::scheduler::Scheduler<SwarmNodeKind, SwarmResourceKind>>,
) -> Result<Vec<hive_jobq::NodeId>> {
sched
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.insert_job(None, |b| {
vec![b.node(SwarmNodeKind::MintBuilderForgeToken).guid()]
})
.map_err(|e| anyhow::anyhow!("{e}"))
}
/// The homeserver agents' own matrix accounts are minted on. Unset is this
/// controller creating agents with no matrix account, which the mint node
/// reports by name rather than `main` refusing to start.
@ -2509,7 +2609,8 @@ fn spawn_matrix_account_backfill(
}
/// Start the forge's periodic passes — the swarm-wide objects, the runner
/// registration token, hives' read tokens, agents' `internal/docs` grants, and
/// registration token, hives' and the builder's read tokens, agents'
/// `internal/docs` grants, and
/// agents' tokens (which also
/// sweeps legacy tokens; see `forge::legacy_tokens`) —
/// once there is a forge client. Lifted out of `main` for
@ -2535,6 +2636,12 @@ fn spawn_forge_workers(
}
});
let sched = Arc::clone(jobq);
forge::builder_account::spawn(Arc::clone(&client), move || {
if let Err(e) = queue_builder_forge_token_mint(&sched) {
tracing::warn!(error = %format!("{e:#}"), "builder forge token: queueing failed");
}
});
let sched = Arc::clone(jobq);
forge::docs_grant::spawn(Arc::clone(&client), move |agents| {
if let Err(e) = queue_docs_grants(&sched, agents) {
tracing::warn!(error = %format!("{e:#}"), "docs grants: queueing failed");
@ -4262,6 +4369,10 @@ mod tests {
"mint_classifier_forge_token",
SwarmNodeKind::MintClassifierForgeToken,
),
(
"mint_builder_forge_token",
SwarmNodeKind::MintBuilderForgeToken,
),
];
for (label, kind) in nodes {
let mut sched = hive_jobq::scheduler::Scheduler::new(