Watch
0
0
Fork
You've already forked hyperhive
0

swarm-builder: scope the forge token to the forge, one build at a time, opt-in

argus's review on #5177: the git credential helper answered every `get`,
so a config PR's own flake inputs could have git send the builder's forge
token to any host. The helper is now configured under
`credential.<forge-url>.helper` and itself answers only an https request
whose host is the forge's; a test runs it as git does and checks that
other hosts, http and non-get operations get nothing.

Builds are bounded: every `build_agent` node needs the one `builder` jobq
resource, so one runs at a time and the rest wait pending, and a newer
head cancels its PR's pending build (a running one finishes and writes on
its own commit). The builder also holds a one-permit semaphore around each
build.

`builder` and `review-classifier`, the forge logins swarm-controller owns,
join nix/reserved-names.nix, with a test tying the list to both constants.

The builder is now opt-in like the CI runner: `allSwarmServices` no longer
turns it on, since it runs unreviewed config PRs through the host's nix
daemon. mara had not answered atlas's question on the PR.
This commit is contained in:
atlas 2026-10-12 22:47:30 +02:00
commit cafa120d77
13 changed files with 360 additions and 70 deletions

View file

@ -14,7 +14,10 @@ Where it runs and how it's enabled: [`services.md`](services.md#builder).
swarm-controller already registers on `agent-configs`. An `opened`, swarm-controller already registers on `agent-configs`. An `opened`,
`reopened` or `synchronized` action on an open PR queues one `build_agent` `reopened` or `synchronized` action on an open PR queues one `build_agent`
node carrying the agent, the head commit and the PR number. An edit, a node carrying the agent, the head commit and the PR number. An edit, a
label or a close builds nothing. label or a close builds nothing. Every `build_agent` node needs the one
`builder` slot, so builds run one at a time and the rest wait pending. A
newer head cancels its PR's pending build; a build already running
finishes and writes its status on its own, older commit.
2. The node sets the status to `pending`, then sends the build as a request on 2. The node sets the status to `pending`, then sends the build as a request on
the queue subject `$SWARM.builder.build` and waits up to 61 minutes for the the queue subject `$SWARM.builder.build` and waits up to 61 minutes for the
reply. reply.
@ -67,7 +70,9 @@ creates, puts in the `hives` team of `agent-configs` (read on every config
repo) and mints a `read:repository` token for, in a five-minute pass. `git` repo) and mints a `read:repository` token for, in a five-minute pass. `git`
gets the token through a credential helper reading the child's environment: gets the token through a credential helper reading the child's environment:
it's in no argument, URL or lock, so no line `nix` prints can carry it into a it's in no argument, URL or lock, so no line `nix` prints can carry it into a
status. The builder holds no write credential; swarm-controller writes the status. The same `git` fetches a config's own inputs, so the builder registers
the helper for the forge's URL only, and the helper answers only an `https`
request for the forge's host: any other host gets nothing. The builder holds no write credential; swarm-controller writes the
status with its own forge token. Details per path: status with its own forge token. Details per path:
[`credentials.md`](credentials.md). [`credentials.md`](credentials.md).

View file

@ -244,14 +244,15 @@ the container and the threat model are in
### Builder ### Builder
The swarm builder, in a `swarm-builder` container on the host that sets The swarm builder, in a `swarm-builder` container on the host that sets
`deploy.builder.enable`. `allSwarmServices` defaults it on wherever the host `deploy.builder.enable`. It doesn't derive from `allSwarmServices`: it
has the builder's store identity, on the same terms as the review classifier. evaluates and builds unreviewed config PRs through its host's nix daemon, so
Set `deploy.builder.enable` directly to override either way. running one is a decision of its own, as for the CI runner.
When an `agent-configs/<agent>` PR opens, reopens or gets a new push, When an `agent-configs/<agent>` PR opens, reopens or gets a new push,
swarm-controller queues a `build_agent` job. The builder builds that config at swarm-controller queues a `build_agent` job. Jobs run one at a time, and a
the PR's head on the agent base its own host runs, through the newer push replaces its PR's job while that job is still waiting. The builder
host's nix daemon, and swarm-controller writes the result as a `swarm-builder` builds that config at the PR's head on the agent base its own host runs,
through the host's nix daemon, and swarm-controller writes the result as a `swarm-builder`
commit status on the head. The job shows on the swarm UI's Jobs page. A red commit status on the head. The job shows on the swarm UI's Jobs page. A red
status blocks nothing: the merge still deploys. status blocks nothing: the merge still deploys.

View file

@ -689,11 +689,13 @@ in
this host's nix daemon, and swarm-controller writes the result as a this host's nix daemon, and swarm-controller writes the result as a
`swarm-builder` commit status on the PR. `swarm-builder` commit status on the PR.
On by default where Not derived from
{option}`services.hyperhive.deploy.allSwarmServices` is set and this {option}`services.hyperhive.deploy.allSwarmServices`: the builder
host has the builder's store identity (the pair evaluates and builds unreviewed config PRs on this host's nix daemon,
{option}`services.hyperhive.deploy.bao.builderClientCertFile` and so running one is a decision of its own, as for the CI runner. It needs
`builderClientKeyFile`, minted on the store's host). the builder's store identity
({option}`services.hyperhive.deploy.bao.builderClientCertFile` and
`builderClientKeyFile`), which the store's host mints.
''; '';
}; };

View file

@ -1,8 +1,9 @@
# The swarm builder, in a `swarm-builder` container on the host that sets # The swarm builder, in a `swarm-builder` container on the host that sets
# `deploy.builder.enable`, which `deploy.allSwarmServices` defaults on where # `deploy.builder.enable`, which nothing turns on for the operator: it runs
# the host has the builder's store identity. It builds agent configs for # unreviewed config PRs through the host's nix daemon. It builds agent configs
# swarm-controller: the controller sends a config PR's head over the queue, # for swarm-controller: the controller sends a config PR's head over the
# and the daemon builds that config on this host's agent base and replies. # queue, and the daemon builds that config on this host's agent base and
# replies.
# docs/swarm/builder.md has the design. # docs/swarm/builder.md has the design.
# #
# ⚠️ Its own network namespace on the bridge, as ./swarm-ci-runner.nix has: a # ⚠️ Its own network namespace on the bridge, as ./swarm-ci-runner.nix has: a

View file

@ -117,11 +117,4 @@ in
&& deployCfg.bao.reviewClassifierClientCertFile != null && deployCfg.bao.reviewClassifierClientCertFile != null
&& deployCfg.bao.reviewClassifierClientKeyFile != null && deployCfg.bao.reviewClassifierClientKeyFile != null
); );
# The swarm builder, defaulted on the same terms as the review classifier.
config.services.hyperhive.deploy.builder.enable = lib.mkDefault (
deployCfg.allSwarmServices
&& deployCfg.bao.builderClientCertFile != null
&& deployCfg.bao.builderClientKeyFile != null
);
} }

View file

@ -33,11 +33,6 @@ let
servicesHere = hiveWithController { deploy.allSwarmServices = true; }; servicesHere = hiveWithController { deploy.allSwarmServices = true; };
servicesBaoElsewhere = hive {
deploy.allSwarmServices = true;
deploy.bao.enable = false;
};
natsHere = hive { natsHere = hive {
swarm.nats.enable = true; swarm.nats.enable = true;
swarm.nats.authPackage = pkgs.emptyDirectory; swarm.nats.authPackage = pkgs.emptyDirectory;
@ -59,16 +54,15 @@ let
ok = !(bare.containers ? swarm-builder) && withBuilder.containers ? swarm-builder; ok = !(bare.containers ? swarm-builder) && withBuilder.containers ? swarm-builder;
} }
{ {
# Its module asserts a store identity that only the store's host mints, # It runs unreviewed config PRs through the host's nix daemon, so hosting
# so with the store elsewhere the default stays off rather than failing # the shared services, even with the store's leaf here, does not turn it
# that host's eval. `nats` is the control that the switch is on at all. # on. `nats` is the control that the switch is on at all.
name = "the shared services turn the builder on with the store's leaf and leave it off without"; name = "hosting the shared services leaves the builder off";
ok = ok =
servicesHere.services.hyperhive.deploy.builder.enable !servicesHere.services.hyperhive.deploy.builder.enable
&& servicesHere.containers ? swarm-builder && !(servicesHere.containers ? swarm-builder)
&& !servicesBaoElsewhere.services.hyperhive.deploy.builder.enable && servicesHere.services.hyperhive.deploy.bao.enable
&& !(servicesBaoElsewhere.containers ? swarm-builder) && servicesHere.services.hyperhive.deploy.nats.enable;
&& servicesBaoElsewhere.services.hyperhive.deploy.nats.enable;
} }
{ {
# A config PR is unreviewed input to an evaluation: the container keeps # A config PR is unreviewed input to an evaluation: the container keeps

View file

@ -472,7 +472,7 @@ let
# that lost a container can't pass by comparing fewer of them. # that lost a container can't pass by comparing fewer of them.
name = "every service container restates its host-side privateNetwork"; name = "every service container restates its host-side privateNetwork";
ok = ok =
lib.length (lib.attrNames serviceContainers) == 12 lib.length (lib.attrNames serviceContainers) == 11
&& lib.all ( && lib.all (
c: c.privateNetwork == (c.config.services.hyperhive.swarmContainer.privateNetwork or null) c: c.privateNetwork == (c.config.services.hyperhive.swarmContainer.privateNetwork or null)
) (lib.attrValues serviceContainers); ) (lib.attrValues serviceContainers);

View file

@ -22,9 +22,9 @@
# here forbids it for both. That is the point: two lists is how they drift. # here forbids it for both. That is the point: two lists is how they drift.
# #
# ⚠️ Every entry must be a value some component actually PRODUCES as a message # ⚠️ Every entry must be a value some component actually PRODUCES as a message
# `from`/`to`, a component name the collector builds pipelines from, or a # `from`/`to`, a component name the collector builds pipelines from, or a forge
# username the forge refuses — not a word that merely looked risky. A name in # username the forge refuses or a swarm service owns — not a word that merely
# here that nothing emits or refuses is a refusal with no failure behind it. # looked risky. A name nothing emits or refuses is a refusal with no failure.
# #
# ⚠️ Matched by EQUALITY. Words forbidden *inside* a hive name live in # ⚠️ Matched by EQUALITY. Words forbidden *inside* a hive name live in
# `./reserved-hive-fragments.nix` — read its header before merging the two. # `./reserved-hive-fragments.nix` — read its header before merging the two.
@ -90,6 +90,13 @@
"repo-avatars" "repo-avatars"
"user" "user"
"v2" "v2"
# ---- forge accounts swarm-controller owns ----------------------------
# Service accounts it creates under these logins (`forge::builder_account`,
# `forge::classifier_account`). An agent of the same name would be refused
# one node into provisioning, and would keep the service from its token.
"builder"
"review-classifier"
] ]
# Deliberately absent, and it is a load-bearing omission: # Deliberately absent, and it is a load-bearing omission:
# #

View file

@ -27,5 +27,6 @@ The container holds one credential: its own store certificate, which nspawn
hands in at start without a copy on disk. With it the service reads its hands in at start without a copy on disk. With it the service reads its
queue client secret at start and its read-only forge token at every build. queue client secret at start and its read-only forge token at every build.
`git` gets the token through a credential helper reading the child's `git` gets the token through a credential helper reading the child's
environment, never in a URL or an argument. Its store role reads exactly environment, never in a URL or an argument, and the helper answers only an
`https` request for the forge's host. Its store role reads exactly
those two paths. those two paths.

View file

@ -48,7 +48,7 @@ async fn main() -> Result<()> {
check_flake: var("CHECK_FLAKE")?, check_flake: var("CHECK_FLAKE")?,
hyperhive: var("HYPERHIVE_FLAKE")?, hyperhive: var("HYPERHIVE_FLAKE")?,
nixpkgs: var("NIXPKGS_FLAKE")?, nixpkgs: var("NIXPKGS_FLAKE")?,
forge_url: var("FORGE_URL")?.trim_end_matches('/').to_owned(), forge_url: forge_url()?,
config_org: var("CONFIG_ORG")?, config_org: var("CONFIG_ORG")?,
deadline: BUILD_DEADLINE, deadline: BUILD_DEADLINE,
}); });
@ -74,14 +74,22 @@ async fn main() -> Result<()> {
.with_context(|| format!("subscribing to {SUBJECT}"))?; .with_context(|| format!("subscribing to {SUBJECT}"))?;
tracing::info!(subject = SUBJECT, "builder: awaiting builds"); tracing::info!(subject = SUBJECT, "builder: awaiting builds");
// One build at a time. swarm-controller already sends one at a time; this
// holds the bound for any request it did not serialise.
let slot = Arc::new(tokio::sync::Semaphore::new(1));
while let Some(msg) = requests.next().await { while let Some(msg) = requests.next().await {
let Some(reply_to) = msg.reply.clone() else { let Some(reply_to) = msg.reply.clone() else {
tracing::warn!("a build arrived with no reply subject; dropping it"); tracing::warn!("a build arrived with no reply subject; dropping it");
continue; continue;
}; };
let queue = queue.clone(); let queue = queue.clone();
let slot = Arc::clone(&slot);
let builder = Arc::clone(&builder); let builder = Arc::clone(&builder);
tokio::spawn(async move { tokio::spawn(async move {
let Ok(_permit) = slot.acquire().await else {
return;
};
let reply = handle(&msg.payload, &builder).await; let reply = handle(&msg.payload, &builder).await;
match serde_json::to_vec(&reply) { match serde_json::to_vec(&reply) {
Ok(body) => { Ok(body) => {
@ -188,3 +196,15 @@ async fn client_secret(client_id: &str) -> Result<String> {
), ),
} }
} }
/// The forge's base URL, which must be `https`: the credential helper offers
/// the token to `https` requests for its host alone.
fn forge_url() -> Result<String> {
let name = format!("{PREFIX}_FORGE_URL");
let url = std::env::var(&name).with_context(|| format!("{name} is unset"))?;
let url = url.trim_end_matches('/');
if !url.starts_with("https://") || url["https://".len()..].contains('/') {
bail!("{name} is {url:?}, not https://<host>[:port]");
}
Ok(url.to_owned())
}

View file

@ -5,6 +5,11 @@
//! token. The token reaches `git` through a credential helper that reads it //! token. The token reaches `git` through a credential helper that reads it
//! from the child's environment: it is in no argument, no URL and no lock, so //! from the child's environment: it is in no argument, no URL and no lock, so
//! no error line `nix` prints can carry it into the reply. //! no error line `nix` prints can carry it into the reply.
//!
//! ⚠️ A config's own flake inputs are fetched by the same `git`, from hosts the
//! config names. The helper is configured for the forge's URL only, and itself
//! answers only an `https` request for the forge's host, so the token never
//! goes anywhere else.
use std::path::PathBuf; use std::path::PathBuf;
use std::process::Stdio; use std::process::Stdio;
@ -20,6 +25,10 @@ const TAIL_BYTES: usize = 4000;
/// The variable the credential helper reads the token from, in the child only. /// The variable the credential helper reads the token from, in the child only.
const TOKEN_VAR: &str = "SWARM_BUILDER_GIT_TOKEN"; const TOKEN_VAR: &str = "SWARM_BUILDER_GIT_TOKEN";
/// The variable holding the one host the helper answers for, in the child
/// only.
const HOST_VAR: &str = "SWARM_BUILDER_GIT_HOST";
/// What every build is run with, fixed at start. /// What every build is run with, fixed at start.
pub struct Builder { pub struct Builder {
/// The `nix` binary. /// The `nix` binary.
@ -30,7 +39,7 @@ pub struct Builder {
pub hyperhive: String, pub hyperhive: String,
/// The nixpkgs source this host was deployed from. /// The nixpkgs source this host was deployed from.
pub nixpkgs: String, pub nixpkgs: String,
/// `https://<forge>`, with no trailing slash. /// `https://<forge>`, with no trailing slash and no path.
pub forge_url: String, pub forge_url: String,
/// The org the config repos are in. /// The org the config repos are in.
pub config_org: String, pub config_org: String,
@ -49,7 +58,11 @@ impl Builder {
.env(TOKEN_VAR, token) .env(TOKEN_VAR, token)
.env("GIT_TERMINAL_PROMPT", "0") .env("GIT_TERMINAL_PROMPT", "0")
.env("GIT_CONFIG_COUNT", "1") .env("GIT_CONFIG_COUNT", "1")
.env("GIT_CONFIG_KEY_0", "credential.helper") .env(HOST_VAR, forge_host(&self.forge_url))
.env(
"GIT_CONFIG_KEY_0",
format!("credential.{}.helper", self.forge_url),
)
.env("GIT_CONFIG_VALUE_0", credential_helper()) .env("GIT_CONFIG_VALUE_0", credential_helper())
.stdin(Stdio::null()) .stdin(Stdio::null())
.stdout(Stdio::piped()) .stdout(Stdio::piped())
@ -97,11 +110,22 @@ impl Builder {
} }
} }
/// A `git` credential helper answering every `get` with the builder's login /// The host part of `forge_url` (`https://<host>[:port]`), as `git` names it
/// and the token from [`TOKEN_VAR`]. /// to a credential helper.
fn forge_host(forge_url: &str) -> &str {
let rest = forge_url.strip_prefix("https://").unwrap_or(forge_url);
rest.split('/').next().unwrap_or(rest)
}
/// A `git` credential helper that answers a `get` with the builder's login and
/// the token from [`TOKEN_VAR`] only when the request is `https` to the host in
/// [`HOST_VAR`], and with nothing otherwise.
fn credential_helper() -> String { fn credential_helper() -> String {
format!( format!(
"!f() {{ test \"$1\" = get && printf 'username=builder\\npassword=%s\\n' \"${TOKEN_VAR}\"; }}; f" "!f() {{ test \"$1\" = get || return 0; p=; h=; \
while IFS='=' read -r k v; do case \"$k\" in '') break ;; protocol) p=$v ;; host) h=$v ;; esac; done; \
test -n \"${HOST_VAR}\" && test \"$p\" = https && test \"$h\" = \"${HOST_VAR}\" \
&& printf 'username=builder\\npassword=%s\\n' \"${TOKEN_VAR}\"; return 0; }}; f"
) )
} }
@ -240,6 +264,61 @@ mod tests {
assert!(args.iter().any(|a| a == "--no-write-lock-file")); assert!(args.iter().any(|a| a == "--no-write-lock-file"));
} }
/// What the helper prints for `op` and a request with `protocol` and
/// `host`, run the way `git` runs a `!` helper.
fn helper_answer(op: &str, protocol: &str, host: &str) -> String {
use std::io::Write as _;
let helper = credential_helper();
let script = format!("{} {op}", helper.trim_start_matches('!'));
let mut child = std::process::Command::new("sh")
.args(["-c", &script])
.env(TOKEN_VAR, "s3cret")
.env(HOST_VAR, "forge.example:3000")
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.spawn()
.expect("sh runs");
write!(
child.stdin.take().expect("piped"),
"protocol={protocol}\nhost={host}\npath=agent-configs/atlas\n\n"
)
.expect("the helper reads its request");
let out = child.wait_with_output().expect("the helper exits");
assert!(out.status.success());
String::from_utf8(out.stdout).expect("utf-8")
}
#[test]
fn the_helper_gives_the_token_to_the_forge_alone() {
assert_eq!(
helper_answer("get", "https", "forge.example:3000"),
"username=builder\npassword=s3cret\n"
);
for (op, protocol, host) in [
("get", "https", "github.com"),
("get", "https", "forge.example"),
("get", "https", "forge.example:3000.evil.example"),
("get", "http", "forge.example:3000"),
("store", "https", "forge.example:3000"),
] {
assert_eq!(
helper_answer(op, protocol, host),
"",
"{op} {protocol}://{host}"
);
}
}
#[test]
fn the_helper_is_configured_for_the_forge_url_only() {
assert_eq!(
forge_host("https://forge.example:3000"),
"forge.example:3000"
);
assert_eq!(forge_host("https://forge.example/sub"), "forge.example");
}
#[test] #[test]
fn the_token_is_in_no_argument() { fn the_token_is_in_no_argument() {
let args = builder().args(&job()).join(" "); let args = builder().args(&job()).join(" ");

View file

@ -4,6 +4,10 @@
//! The status is the check's only record besides the node itself: nothing is //! The status is the check's only record besides the node itself: nothing is
//! pushed, and the closure the builder made stays in its host's store. A red //! pushed, and the closure the builder made stays in its host's store. A red
//! status blocks nothing; the merge deploys as before. //! status blocks nothing; the merge deploys as before.
//!
//! Builds share one `SwarmResourceKind::Builder` slot, so one runs at a time,
//! and a newer push replaces its PR's build while that build is still
//! pending (`queue_build`).
use std::future::Future; use std::future::Future;
@ -227,6 +231,59 @@ mod tests {
); );
} }
/// A superseded build that finishes after the newer head's writes only on
/// its own commit, so the newer head keeps its status.
#[tokio::test]
async fn a_late_superseded_build_writes_only_on_its_own_commit() {
struct ByRevQueue;
impl BuildSink for ByRevQueue {
async fn build(&self, request: Vec<u8>) -> Result<Reply, String> {
let sent: BuildAgent = serde_json::from_slice(&request).expect("a build request");
Ok(if sent.rev == "old" {
Reply::Failed {
error_tail: "error: boom".to_owned(),
}
} else {
Reply::Built {
out_path: "/nix/store/new".to_owned(),
}
})
}
}
#[derive(Default)]
struct ByRev(Mutex<Vec<(String, CommitStatusState)>>);
impl StatusSink for ByRev {
async fn set(
&self,
_: &str,
rev: &str,
state: CommitStatusState,
_: &str,
) -> anyhow::Result<()> {
self.0.lock().unwrap().push((rev.to_owned(), state));
Ok(())
}
}
let statuses = ByRev::default();
let at = |rev: &str| BuildAgent {
rev: rev.to_owned(),
..job()
};
run(Some(&ByRevQueue), Some(&statuses), at("new")).await;
run(Some(&ByRevQueue), Some(&statuses), at("old")).await;
let written = statuses.0.into_inner().unwrap();
let last_on = |rev: &str| {
written
.iter()
.rev()
.find(|(r, _)| r == rev)
.map(|(_, s)| *s)
};
assert_eq!(last_on("new"), Some(CommitStatusState::Success));
assert_eq!(last_on("old"), Some(CommitStatusState::Failure));
}
#[tokio::test] #[tokio::test]
async fn no_builder_is_an_error_status_not_a_failure() { async fn no_builder_is_an_error_status_not_a_failure() {
let (outcome, states) = let (outcome, states) =

View file

@ -254,14 +254,20 @@ impl hive_jobq_wire::WireNode for SwarmNodeKind {
} }
} }
/// Placeholder resource name — same rationale and same "no variants until a /// A named counting semaphore a node acquires before it starts. Every name
/// real node needs one" shape as [`SwarmNodeKind`]. /// has capacity 1: the scheduler's default, which no caller overrides.
#[derive(Clone, Debug, PartialEq, Eq, Hash)] #[derive(Clone, Debug, PartialEq, Eq, Hash)]
enum SwarmResourceKind {} enum SwarmResourceKind {
/// The swarm builder: one `BuildAgent` node runs at a time, and the rest
/// wait pending, where a newer push can replace them.
Builder,
}
impl hive_jobq_wire::WireResource for SwarmResourceKind { impl hive_jobq_wire::WireResource for SwarmResourceKind {
fn name(&self) -> String { fn name(&self) -> String {
match *self {} match self {
SwarmResourceKind::Builder => "builder".to_owned(),
}
} }
} }
@ -1960,29 +1966,58 @@ fn claiming_hives(
/// whichever hive the agent runs: the check builds on the builder's own /// whichever hive the agent runs: the check builds on the builder's own
/// agent base. /// agent base.
fn queue_config_build(state: &AppState, changed: config_pr::ChangedConfigPr) { fn queue_config_build(state: &AppState, changed: config_pr::ChangedConfigPr) {
let config_pr::ChangedConfigPr { agent, rev, pr } = changed; let (agent, rev, pr) = (changed.agent.clone(), changed.rev.clone(), changed.pr);
let inserted = state let queued = queue_build(
.jobq &mut state
.lock() .jobq
.unwrap_or_else(std::sync::PoisonError::into_inner) .lock()
.insert_job(None, |b| { .unwrap_or_else(std::sync::PoisonError::into_inner),
vec![ changed,
b.node(SwarmNodeKind::BuildAgent { );
agent: agent.clone(), match queued {
rev: rev.clone(), Ok(superseded) => {
pr, tracing::info!(%agent, %rev, pr, superseded, "config-pr head: build queued");
}) }
.guid(),
]
});
match inserted {
Ok(_) => tracing::info!(%agent, %rev, pr, "config-pr head: build queued"),
Err(e) => { Err(e) => {
tracing::warn!(%agent, %rev, pr, error = %e, "config-pr head: queueing the build failed"); tracing::warn!(%agent, %rev, pr, error = %e, "config-pr head: queueing the build failed");
} }
} }
} }
/// Cancel every pending build of the same PR, then insert the new one behind
/// [`SwarmResourceKind::Builder`]. A build already running is left to finish:
/// its status lands on its own commit, not on the new head. Returns how many
/// pending builds were cancelled.
fn queue_build(
sched: &mut hive_jobq::scheduler::Scheduler<SwarmNodeKind, SwarmResourceKind>,
changed: config_pr::ChangedConfigPr,
) -> Result<usize> {
let config_pr::ChangedConfigPr { agent, rev, pr } = changed;
let stale: Vec<hive_jobq::NodeId> = sched
.graph()
.nodes()
.filter(|n| n.state == hive_jobq::State::Pending)
.filter(|n| {
matches!(&n.payload, SwarmNodeKind::BuildAgent { agent: a, pr: p, .. } if *a == agent && *p == pr)
})
.map(|n| n.id)
.collect();
let superseded = stale
.into_iter()
.filter(|&id| sched.cancel_node(id))
.count();
sched
.insert_job(None, |b| {
vec![
b.node(SwarmNodeKind::BuildAgent { agent, rev, pr })
.needs(SwarmResourceKind::Builder)
.guid(),
]
})
.map_err(|e| anyhow::anyhow!("{e}"))?;
Ok(superseded)
}
/// Queue the deploy of a merged config PR on the one hive that runs its agent. /// Queue the deploy of a merged config PR on the one hive that runs its agent.
/// ///
/// The hive is found by scanning every hive's wanted state, since nothing /// The hive is found by scanning every hive's wanted state, since nothing
@ -4147,6 +4182,101 @@ mod tests {
} }
} }
fn changed(agent: &str, rev: &str, pr: u64) -> crate::config_pr::ChangedConfigPr {
crate::config_pr::ChangedConfigPr {
agent: agent.to_owned(),
rev: rev.to_owned(),
pr,
}
}
fn build_state(
sched: &hive_jobq::scheduler::Scheduler<SwarmNodeKind, super::SwarmResourceKind>,
rev: &str,
) -> hive_jobq::State {
sched
.graph()
.nodes()
.find(|n| matches!(&n.payload, SwarmNodeKind::BuildAgent { rev: r, .. } if r == rev))
.map(|n| n.state)
.expect("a build of that rev")
}
/// A newer head of the same PR replaces its pending build; another PR's
/// build, and the same PR's running build, are left alone.
#[tokio::test]
async fn a_newer_push_replaces_the_prs_pending_build_only() {
let sched =
std::sync::Arc::new(std::sync::Mutex::new(hive_jobq::scheduler::Scheduler::new(
hive_jobq::Graph::new(),
hive_jobq::resources::ResourceTable::new(),
)));
let queue = |c| super::queue_build(&mut sched.lock().unwrap(), c).expect("queued");
assert_eq!(queue(changed("atlas", "r1", 3)), 0);
// r1 takes the builder and stays running: its future is never awaited.
let _running = hive_jobq::scheduler::Scheduler::claim_next(&sched, |_, _, b| async move {
(b, hive_jobq::scheduler::Outcome::Done)
})
.expect("r1 is runnable");
assert_eq!(
queue(changed("atlas", "r2", 3)),
0,
"r1 is running, not pending"
);
assert_eq!(queue(changed("argus", "o1", 4)), 0);
assert_eq!(queue(changed("atlas", "r3", 3)), 1, "r2 was pending");
let guard = sched.lock().unwrap();
assert_eq!(build_state(&guard, "r1"), hive_jobq::State::Running);
assert_eq!(build_state(&guard, "r2"), hive_jobq::State::Cancelled);
assert_eq!(build_state(&guard, "r3"), hive_jobq::State::Pending);
assert_eq!(build_state(&guard, "o1"), hive_jobq::State::Pending);
}
/// Builds share one builder slot: a second build is not claimable while
/// the first runs, and is once it has finished.
#[tokio::test]
async fn one_build_runs_at_a_time() {
let sched =
std::sync::Arc::new(std::sync::Mutex::new(hive_jobq::scheduler::Scheduler::new(
hive_jobq::Graph::new(),
hive_jobq::resources::ResourceTable::new(),
)));
for c in [changed("atlas", "a1", 1), changed("argus", "b1", 2)] {
super::queue_build(&mut sched.lock().unwrap(), c).expect("queued");
}
let run = |sched| {
hive_jobq::scheduler::Scheduler::claim_next(sched, |_, _, b| async move {
(b, hive_jobq::scheduler::Outcome::Done)
})
};
let first = run(&sched).expect("one build is runnable");
assert!(
run(&sched).is_none(),
"a second build started beside the first"
);
first.await.1.expect("no growth");
run(&sched)
.expect("the second build runs once the first is done")
.await
.1
.expect("no growth");
}
#[test]
fn a_new_agent_named_like_a_service_forge_account_is_refused() {
let raw = nix_reserved();
for name in [
crate::forge::builder_account::LOGIN,
crate::forge::classifier_account::LOGIN,
] {
let mut warnings = Vec::new();
let broken = super::broken_name_rules(name, Some(&raw), &mut warnings);
let err = super::name_verdict(name, &broken, Ok(false), &mut warnings).expect_err(name);
assert!(err.contains("nix/reserved-names.nix"), "{name:?}: {err}");
}
}
/// One case per shape rule, each named in the refusal. Checked against a /// One case per shape rule, each named in the refusal. Checked against a
/// list without them, so the literal half cannot carry this test. /// list without them, so the literal half cannot carry this test.
#[test] #[test]