swarm-builder: scope the forge token to the forge, one build at a time, opt-in
argus's review on #5177: the git credential helper answered every `get`, so a config PR's own flake inputs could have git send the builder's forge token to any host. The helper is now configured under `credential.<forge-url>.helper` and itself answers only an https request whose host is the forge's; a test runs it as git does and checks that other hosts, http and non-get operations get nothing. Builds are bounded: every `build_agent` node needs the one `builder` jobq resource, so one runs at a time and the rest wait pending, and a newer head cancels its PR's pending build (a running one finishes and writes on its own commit). The builder also holds a one-permit semaphore around each build. `builder` and `review-classifier`, the forge logins swarm-controller owns, join nix/reserved-names.nix, with a test tying the list to both constants. The builder is now opt-in like the CI runner: `allSwarmServices` no longer turns it on, since it runs unreviewed config PRs through the host's nix daemon. mara had not answered atlas's question on the PR.
This commit is contained in:
parent
b68b74d687
commit
cafa120d77
13 changed files with 360 additions and 70 deletions
|
|
@ -14,7 +14,10 @@ Where it runs and how it's enabled: [`services.md`](services.md#builder).
|
|||
swarm-controller already registers on `agent-configs`. An `opened`,
|
||||
`reopened` or `synchronized` action on an open PR queues one `build_agent`
|
||||
node carrying the agent, the head commit and the PR number. An edit, a
|
||||
label or a close builds nothing.
|
||||
label or a close builds nothing. Every `build_agent` node needs the one
|
||||
`builder` slot, so builds run one at a time and the rest wait pending. A
|
||||
newer head cancels its PR's pending build; a build already running
|
||||
finishes and writes its status on its own, older commit.
|
||||
2. The node sets the status to `pending`, then sends the build as a request on
|
||||
the queue subject `$SWARM.builder.build` and waits up to 61 minutes for the
|
||||
reply.
|
||||
|
|
@ -67,7 +70,9 @@ creates, puts in the `hives` team of `agent-configs` (read on every config
|
|||
repo) and mints a `read:repository` token for, in a five-minute pass. `git`
|
||||
gets the token through a credential helper reading the child's environment:
|
||||
it's in no argument, URL or lock, so no line `nix` prints can carry it into a
|
||||
status. The builder holds no write credential; swarm-controller writes the
|
||||
status. The same `git` fetches a config's own inputs, so the builder registers
|
||||
the helper for the forge's URL only, and the helper answers only an `https`
|
||||
request for the forge's host: any other host gets nothing. The builder holds no write credential; swarm-controller writes the
|
||||
status with its own forge token. Details per path:
|
||||
[`credentials.md`](credentials.md).
|
||||
|
||||
|
|
|
|||
|
|
@ -244,14 +244,15 @@ the container and the threat model are in
|
|||
### Builder
|
||||
|
||||
The swarm builder, in a `swarm-builder` container on the host that sets
|
||||
`deploy.builder.enable`. `allSwarmServices` defaults it on wherever the host
|
||||
has the builder's store identity, on the same terms as the review classifier.
|
||||
Set `deploy.builder.enable` directly to override either way.
|
||||
`deploy.builder.enable`. It doesn't derive from `allSwarmServices`: it
|
||||
evaluates and builds unreviewed config PRs through its host's nix daemon, so
|
||||
running one is a decision of its own, as for the CI runner.
|
||||
|
||||
When an `agent-configs/<agent>` PR opens, reopens or gets a new push,
|
||||
swarm-controller queues a `build_agent` job. The builder builds that config at
|
||||
the PR's head on the agent base its own host runs, through the
|
||||
host's nix daemon, and swarm-controller writes the result as a `swarm-builder`
|
||||
swarm-controller queues a `build_agent` job. Jobs run one at a time, and a
|
||||
newer push replaces its PR's job while that job is still waiting. The builder
|
||||
builds that config at the PR's head on the agent base its own host runs,
|
||||
through the host's nix daemon, and swarm-controller writes the result as a `swarm-builder`
|
||||
commit status on the head. The job shows on the swarm UI's Jobs page. A red
|
||||
status blocks nothing: the merge still deploys.
|
||||
|
||||
|
|
|
|||
|
|
@ -689,11 +689,13 @@ in
|
|||
this host's nix daemon, and swarm-controller writes the result as a
|
||||
`swarm-builder` commit status on the PR.
|
||||
|
||||
On by default where
|
||||
{option}`services.hyperhive.deploy.allSwarmServices` is set and this
|
||||
host has the builder's store identity (the pair
|
||||
{option}`services.hyperhive.deploy.bao.builderClientCertFile` and
|
||||
`builderClientKeyFile`, minted on the store's host).
|
||||
Not derived from
|
||||
{option}`services.hyperhive.deploy.allSwarmServices`: the builder
|
||||
evaluates and builds unreviewed config PRs on this host's nix daemon,
|
||||
so running one is a decision of its own, as for the CI runner. It needs
|
||||
the builder's store identity
|
||||
({option}`services.hyperhive.deploy.bao.builderClientCertFile` and
|
||||
`builderClientKeyFile`), which the store's host mints.
|
||||
'';
|
||||
};
|
||||
|
||||
|
|
|
|||
|
|
@ -1,8 +1,9 @@
|
|||
# The swarm builder, in a `swarm-builder` container on the host that sets
|
||||
# `deploy.builder.enable`, which `deploy.allSwarmServices` defaults on where
|
||||
# the host has the builder's store identity. It builds agent configs for
|
||||
# swarm-controller: the controller sends a config PR's head over the queue,
|
||||
# and the daemon builds that config on this host's agent base and replies.
|
||||
# `deploy.builder.enable`, which nothing turns on for the operator: it runs
|
||||
# unreviewed config PRs through the host's nix daemon. It builds agent configs
|
||||
# for swarm-controller: the controller sends a config PR's head over the
|
||||
# queue, and the daemon builds that config on this host's agent base and
|
||||
# replies.
|
||||
# docs/swarm/builder.md has the design.
|
||||
#
|
||||
# ⚠️ Its own network namespace on the bridge, as ./swarm-ci-runner.nix has: a
|
||||
|
|
|
|||
|
|
@ -117,11 +117,4 @@ in
|
|||
&& deployCfg.bao.reviewClassifierClientCertFile != null
|
||||
&& deployCfg.bao.reviewClassifierClientKeyFile != null
|
||||
);
|
||||
|
||||
# The swarm builder, defaulted on the same terms as the review classifier.
|
||||
config.services.hyperhive.deploy.builder.enable = lib.mkDefault (
|
||||
deployCfg.allSwarmServices
|
||||
&& deployCfg.bao.builderClientCertFile != null
|
||||
&& deployCfg.bao.builderClientKeyFile != null
|
||||
);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -33,11 +33,6 @@ let
|
|||
|
||||
servicesHere = hiveWithController { deploy.allSwarmServices = true; };
|
||||
|
||||
servicesBaoElsewhere = hive {
|
||||
deploy.allSwarmServices = true;
|
||||
deploy.bao.enable = false;
|
||||
};
|
||||
|
||||
natsHere = hive {
|
||||
swarm.nats.enable = true;
|
||||
swarm.nats.authPackage = pkgs.emptyDirectory;
|
||||
|
|
@ -59,16 +54,15 @@ let
|
|||
ok = !(bare.containers ? swarm-builder) && withBuilder.containers ? swarm-builder;
|
||||
}
|
||||
{
|
||||
# Its module asserts a store identity that only the store's host mints,
|
||||
# so with the store elsewhere the default stays off rather than failing
|
||||
# that host's eval. `nats` is the control that the switch is on at all.
|
||||
name = "the shared services turn the builder on with the store's leaf and leave it off without";
|
||||
# It runs unreviewed config PRs through the host's nix daemon, so hosting
|
||||
# the shared services, even with the store's leaf here, does not turn it
|
||||
# on. `nats` is the control that the switch is on at all.
|
||||
name = "hosting the shared services leaves the builder off";
|
||||
ok =
|
||||
servicesHere.services.hyperhive.deploy.builder.enable
|
||||
&& servicesHere.containers ? swarm-builder
|
||||
&& !servicesBaoElsewhere.services.hyperhive.deploy.builder.enable
|
||||
&& !(servicesBaoElsewhere.containers ? swarm-builder)
|
||||
&& servicesBaoElsewhere.services.hyperhive.deploy.nats.enable;
|
||||
!servicesHere.services.hyperhive.deploy.builder.enable
|
||||
&& !(servicesHere.containers ? swarm-builder)
|
||||
&& servicesHere.services.hyperhive.deploy.bao.enable
|
||||
&& servicesHere.services.hyperhive.deploy.nats.enable;
|
||||
}
|
||||
{
|
||||
# A config PR is unreviewed input to an evaluation: the container keeps
|
||||
|
|
|
|||
|
|
@ -472,7 +472,7 @@ let
|
|||
# that lost a container can't pass by comparing fewer of them.
|
||||
name = "every service container restates its host-side privateNetwork";
|
||||
ok =
|
||||
lib.length (lib.attrNames serviceContainers) == 12
|
||||
lib.length (lib.attrNames serviceContainers) == 11
|
||||
&& lib.all (
|
||||
c: c.privateNetwork == (c.config.services.hyperhive.swarmContainer.privateNetwork or null)
|
||||
) (lib.attrValues serviceContainers);
|
||||
|
|
|
|||
|
|
@ -22,9 +22,9 @@
|
|||
# here forbids it for both. That is the point: two lists is how they drift.
|
||||
#
|
||||
# ⚠️ Every entry must be a value some component actually PRODUCES as a message
|
||||
# `from`/`to`, a component name the collector builds pipelines from, or a
|
||||
# username the forge refuses — not a word that merely looked risky. A name in
|
||||
# here that nothing emits or refuses is a refusal with no failure behind it.
|
||||
# `from`/`to`, a component name the collector builds pipelines from, or a forge
|
||||
# username the forge refuses or a swarm service owns — not a word that merely
|
||||
# looked risky. A name nothing emits or refuses is a refusal with no failure.
|
||||
#
|
||||
# ⚠️ Matched by EQUALITY. Words forbidden *inside* a hive name live in
|
||||
# `./reserved-hive-fragments.nix` — read its header before merging the two.
|
||||
|
|
@ -90,6 +90,13 @@
|
|||
"repo-avatars"
|
||||
"user"
|
||||
"v2"
|
||||
|
||||
# ---- forge accounts swarm-controller owns ----------------------------
|
||||
# Service accounts it creates under these logins (`forge::builder_account`,
|
||||
# `forge::classifier_account`). An agent of the same name would be refused
|
||||
# one node into provisioning, and would keep the service from its token.
|
||||
"builder"
|
||||
"review-classifier"
|
||||
]
|
||||
# Deliberately absent, and it is a load-bearing omission:
|
||||
#
|
||||
|
|
|
|||
|
|
@ -27,5 +27,6 @@ The container holds one credential: its own store certificate, which nspawn
|
|||
hands in at start without a copy on disk. With it the service reads its
|
||||
queue client secret at start and its read-only forge token at every build.
|
||||
`git` gets the token through a credential helper reading the child's
|
||||
environment, never in a URL or an argument. Its store role reads exactly
|
||||
environment, never in a URL or an argument, and the helper answers only an
|
||||
`https` request for the forge's host. Its store role reads exactly
|
||||
those two paths.
|
||||
|
|
|
|||
|
|
@ -48,7 +48,7 @@ async fn main() -> Result<()> {
|
|||
check_flake: var("CHECK_FLAKE")?,
|
||||
hyperhive: var("HYPERHIVE_FLAKE")?,
|
||||
nixpkgs: var("NIXPKGS_FLAKE")?,
|
||||
forge_url: var("FORGE_URL")?.trim_end_matches('/').to_owned(),
|
||||
forge_url: forge_url()?,
|
||||
config_org: var("CONFIG_ORG")?,
|
||||
deadline: BUILD_DEADLINE,
|
||||
});
|
||||
|
|
@ -74,14 +74,22 @@ async fn main() -> Result<()> {
|
|||
.with_context(|| format!("subscribing to {SUBJECT}"))?;
|
||||
tracing::info!(subject = SUBJECT, "builder: awaiting builds");
|
||||
|
||||
// One build at a time. swarm-controller already sends one at a time; this
|
||||
// holds the bound for any request it did not serialise.
|
||||
let slot = Arc::new(tokio::sync::Semaphore::new(1));
|
||||
|
||||
while let Some(msg) = requests.next().await {
|
||||
let Some(reply_to) = msg.reply.clone() else {
|
||||
tracing::warn!("a build arrived with no reply subject; dropping it");
|
||||
continue;
|
||||
};
|
||||
let queue = queue.clone();
|
||||
let slot = Arc::clone(&slot);
|
||||
let builder = Arc::clone(&builder);
|
||||
tokio::spawn(async move {
|
||||
let Ok(_permit) = slot.acquire().await else {
|
||||
return;
|
||||
};
|
||||
let reply = handle(&msg.payload, &builder).await;
|
||||
match serde_json::to_vec(&reply) {
|
||||
Ok(body) => {
|
||||
|
|
@ -188,3 +196,15 @@ async fn client_secret(client_id: &str) -> Result<String> {
|
|||
),
|
||||
}
|
||||
}
|
||||
|
||||
/// The forge's base URL, which must be `https`: the credential helper offers
|
||||
/// the token to `https` requests for its host alone.
|
||||
fn forge_url() -> Result<String> {
|
||||
let name = format!("{PREFIX}_FORGE_URL");
|
||||
let url = std::env::var(&name).with_context(|| format!("{name} is unset"))?;
|
||||
let url = url.trim_end_matches('/');
|
||||
if !url.starts_with("https://") || url["https://".len()..].contains('/') {
|
||||
bail!("{name} is {url:?}, not https://<host>[:port]");
|
||||
}
|
||||
Ok(url.to_owned())
|
||||
}
|
||||
|
|
|
|||
|
|
@ -5,6 +5,11 @@
|
|||
//! token. The token reaches `git` through a credential helper that reads it
|
||||
//! from the child's environment: it is in no argument, no URL and no lock, so
|
||||
//! no error line `nix` prints can carry it into the reply.
|
||||
//!
|
||||
//! ⚠️ A config's own flake inputs are fetched by the same `git`, from hosts the
|
||||
//! config names. The helper is configured for the forge's URL only, and itself
|
||||
//! answers only an `https` request for the forge's host, so the token never
|
||||
//! goes anywhere else.
|
||||
|
||||
use std::path::PathBuf;
|
||||
use std::process::Stdio;
|
||||
|
|
@ -20,6 +25,10 @@ const TAIL_BYTES: usize = 4000;
|
|||
/// The variable the credential helper reads the token from, in the child only.
|
||||
const TOKEN_VAR: &str = "SWARM_BUILDER_GIT_TOKEN";
|
||||
|
||||
/// The variable holding the one host the helper answers for, in the child
|
||||
/// only.
|
||||
const HOST_VAR: &str = "SWARM_BUILDER_GIT_HOST";
|
||||
|
||||
/// What every build is run with, fixed at start.
|
||||
pub struct Builder {
|
||||
/// The `nix` binary.
|
||||
|
|
@ -30,7 +39,7 @@ pub struct Builder {
|
|||
pub hyperhive: String,
|
||||
/// The nixpkgs source this host was deployed from.
|
||||
pub nixpkgs: String,
|
||||
/// `https://<forge>`, with no trailing slash.
|
||||
/// `https://<forge>`, with no trailing slash and no path.
|
||||
pub forge_url: String,
|
||||
/// The org the config repos are in.
|
||||
pub config_org: String,
|
||||
|
|
@ -49,7 +58,11 @@ impl Builder {
|
|||
.env(TOKEN_VAR, token)
|
||||
.env("GIT_TERMINAL_PROMPT", "0")
|
||||
.env("GIT_CONFIG_COUNT", "1")
|
||||
.env("GIT_CONFIG_KEY_0", "credential.helper")
|
||||
.env(HOST_VAR, forge_host(&self.forge_url))
|
||||
.env(
|
||||
"GIT_CONFIG_KEY_0",
|
||||
format!("credential.{}.helper", self.forge_url),
|
||||
)
|
||||
.env("GIT_CONFIG_VALUE_0", credential_helper())
|
||||
.stdin(Stdio::null())
|
||||
.stdout(Stdio::piped())
|
||||
|
|
@ -97,11 +110,22 @@ impl Builder {
|
|||
}
|
||||
}
|
||||
|
||||
/// A `git` credential helper answering every `get` with the builder's login
|
||||
/// and the token from [`TOKEN_VAR`].
|
||||
/// The host part of `forge_url` (`https://<host>[:port]`), as `git` names it
|
||||
/// to a credential helper.
|
||||
fn forge_host(forge_url: &str) -> &str {
|
||||
let rest = forge_url.strip_prefix("https://").unwrap_or(forge_url);
|
||||
rest.split('/').next().unwrap_or(rest)
|
||||
}
|
||||
|
||||
/// A `git` credential helper that answers a `get` with the builder's login and
|
||||
/// the token from [`TOKEN_VAR`] only when the request is `https` to the host in
|
||||
/// [`HOST_VAR`], and with nothing otherwise.
|
||||
fn credential_helper() -> String {
|
||||
format!(
|
||||
"!f() {{ test \"$1\" = get && printf 'username=builder\\npassword=%s\\n' \"${TOKEN_VAR}\"; }}; f"
|
||||
"!f() {{ test \"$1\" = get || return 0; p=; h=; \
|
||||
while IFS='=' read -r k v; do case \"$k\" in '') break ;; protocol) p=$v ;; host) h=$v ;; esac; done; \
|
||||
test -n \"${HOST_VAR}\" && test \"$p\" = https && test \"$h\" = \"${HOST_VAR}\" \
|
||||
&& printf 'username=builder\\npassword=%s\\n' \"${TOKEN_VAR}\"; return 0; }}; f"
|
||||
)
|
||||
}
|
||||
|
||||
|
|
@ -240,6 +264,61 @@ mod tests {
|
|||
assert!(args.iter().any(|a| a == "--no-write-lock-file"));
|
||||
}
|
||||
|
||||
/// What the helper prints for `op` and a request with `protocol` and
|
||||
/// `host`, run the way `git` runs a `!` helper.
|
||||
fn helper_answer(op: &str, protocol: &str, host: &str) -> String {
|
||||
use std::io::Write as _;
|
||||
|
||||
let helper = credential_helper();
|
||||
let script = format!("{} {op}", helper.trim_start_matches('!'));
|
||||
let mut child = std::process::Command::new("sh")
|
||||
.args(["-c", &script])
|
||||
.env(TOKEN_VAR, "s3cret")
|
||||
.env(HOST_VAR, "forge.example:3000")
|
||||
.stdin(Stdio::piped())
|
||||
.stdout(Stdio::piped())
|
||||
.spawn()
|
||||
.expect("sh runs");
|
||||
write!(
|
||||
child.stdin.take().expect("piped"),
|
||||
"protocol={protocol}\nhost={host}\npath=agent-configs/atlas\n\n"
|
||||
)
|
||||
.expect("the helper reads its request");
|
||||
let out = child.wait_with_output().expect("the helper exits");
|
||||
assert!(out.status.success());
|
||||
String::from_utf8(out.stdout).expect("utf-8")
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_helper_gives_the_token_to_the_forge_alone() {
|
||||
assert_eq!(
|
||||
helper_answer("get", "https", "forge.example:3000"),
|
||||
"username=builder\npassword=s3cret\n"
|
||||
);
|
||||
for (op, protocol, host) in [
|
||||
("get", "https", "github.com"),
|
||||
("get", "https", "forge.example"),
|
||||
("get", "https", "forge.example:3000.evil.example"),
|
||||
("get", "http", "forge.example:3000"),
|
||||
("store", "https", "forge.example:3000"),
|
||||
] {
|
||||
assert_eq!(
|
||||
helper_answer(op, protocol, host),
|
||||
"",
|
||||
"{op} {protocol}://{host}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_helper_is_configured_for_the_forge_url_only() {
|
||||
assert_eq!(
|
||||
forge_host("https://forge.example:3000"),
|
||||
"forge.example:3000"
|
||||
);
|
||||
assert_eq!(forge_host("https://forge.example/sub"), "forge.example");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_token_is_in_no_argument() {
|
||||
let args = builder().args(&job()).join(" ");
|
||||
|
|
|
|||
|
|
@ -4,6 +4,10 @@
|
|||
//! The status is the check's only record besides the node itself: nothing is
|
||||
//! pushed, and the closure the builder made stays in its host's store. A red
|
||||
//! status blocks nothing; the merge deploys as before.
|
||||
//!
|
||||
//! Builds share one `SwarmResourceKind::Builder` slot, so one runs at a time,
|
||||
//! and a newer push replaces its PR's build while that build is still
|
||||
//! pending (`queue_build`).
|
||||
|
||||
use std::future::Future;
|
||||
|
||||
|
|
@ -227,6 +231,59 @@ mod tests {
|
|||
);
|
||||
}
|
||||
|
||||
/// A superseded build that finishes after the newer head's writes only on
|
||||
/// its own commit, so the newer head keeps its status.
|
||||
#[tokio::test]
|
||||
async fn a_late_superseded_build_writes_only_on_its_own_commit() {
|
||||
struct ByRevQueue;
|
||||
impl BuildSink for ByRevQueue {
|
||||
async fn build(&self, request: Vec<u8>) -> Result<Reply, String> {
|
||||
let sent: BuildAgent = serde_json::from_slice(&request).expect("a build request");
|
||||
Ok(if sent.rev == "old" {
|
||||
Reply::Failed {
|
||||
error_tail: "error: boom".to_owned(),
|
||||
}
|
||||
} else {
|
||||
Reply::Built {
|
||||
out_path: "/nix/store/new".to_owned(),
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
#[derive(Default)]
|
||||
struct ByRev(Mutex<Vec<(String, CommitStatusState)>>);
|
||||
impl StatusSink for ByRev {
|
||||
async fn set(
|
||||
&self,
|
||||
_: &str,
|
||||
rev: &str,
|
||||
state: CommitStatusState,
|
||||
_: &str,
|
||||
) -> anyhow::Result<()> {
|
||||
self.0.lock().unwrap().push((rev.to_owned(), state));
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
let statuses = ByRev::default();
|
||||
let at = |rev: &str| BuildAgent {
|
||||
rev: rev.to_owned(),
|
||||
..job()
|
||||
};
|
||||
run(Some(&ByRevQueue), Some(&statuses), at("new")).await;
|
||||
run(Some(&ByRevQueue), Some(&statuses), at("old")).await;
|
||||
let written = statuses.0.into_inner().unwrap();
|
||||
let last_on = |rev: &str| {
|
||||
written
|
||||
.iter()
|
||||
.rev()
|
||||
.find(|(r, _)| r == rev)
|
||||
.map(|(_, s)| *s)
|
||||
};
|
||||
assert_eq!(last_on("new"), Some(CommitStatusState::Success));
|
||||
assert_eq!(last_on("old"), Some(CommitStatusState::Failure));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn no_builder_is_an_error_status_not_a_failure() {
|
||||
let (outcome, states) =
|
||||
|
|
|
|||
|
|
@ -254,14 +254,20 @@ impl hive_jobq_wire::WireNode for SwarmNodeKind {
|
|||
}
|
||||
}
|
||||
|
||||
/// Placeholder resource name — same rationale and same "no variants until a
|
||||
/// real node needs one" shape as [`SwarmNodeKind`].
|
||||
/// A named counting semaphore a node acquires before it starts. Every name
|
||||
/// has capacity 1: the scheduler's default, which no caller overrides.
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Hash)]
|
||||
enum SwarmResourceKind {}
|
||||
enum SwarmResourceKind {
|
||||
/// The swarm builder: one `BuildAgent` node runs at a time, and the rest
|
||||
/// wait pending, where a newer push can replace them.
|
||||
Builder,
|
||||
}
|
||||
|
||||
impl hive_jobq_wire::WireResource for SwarmResourceKind {
|
||||
fn name(&self) -> String {
|
||||
match *self {}
|
||||
match self {
|
||||
SwarmResourceKind::Builder => "builder".to_owned(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -1960,29 +1966,58 @@ fn claiming_hives(
|
|||
/// whichever hive the agent runs: the check builds on the builder's own
|
||||
/// agent base.
|
||||
fn queue_config_build(state: &AppState, changed: config_pr::ChangedConfigPr) {
|
||||
let config_pr::ChangedConfigPr { agent, rev, pr } = changed;
|
||||
let inserted = state
|
||||
.jobq
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner)
|
||||
.insert_job(None, |b| {
|
||||
vec![
|
||||
b.node(SwarmNodeKind::BuildAgent {
|
||||
agent: agent.clone(),
|
||||
rev: rev.clone(),
|
||||
pr,
|
||||
})
|
||||
.guid(),
|
||||
]
|
||||
});
|
||||
match inserted {
|
||||
Ok(_) => tracing::info!(%agent, %rev, pr, "config-pr head: build queued"),
|
||||
let (agent, rev, pr) = (changed.agent.clone(), changed.rev.clone(), changed.pr);
|
||||
let queued = queue_build(
|
||||
&mut state
|
||||
.jobq
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner),
|
||||
changed,
|
||||
);
|
||||
match queued {
|
||||
Ok(superseded) => {
|
||||
tracing::info!(%agent, %rev, pr, superseded, "config-pr head: build queued");
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::warn!(%agent, %rev, pr, error = %e, "config-pr head: queueing the build failed");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Cancel every pending build of the same PR, then insert the new one behind
|
||||
/// [`SwarmResourceKind::Builder`]. A build already running is left to finish:
|
||||
/// its status lands on its own commit, not on the new head. Returns how many
|
||||
/// pending builds were cancelled.
|
||||
fn queue_build(
|
||||
sched: &mut hive_jobq::scheduler::Scheduler<SwarmNodeKind, SwarmResourceKind>,
|
||||
changed: config_pr::ChangedConfigPr,
|
||||
) -> Result<usize> {
|
||||
let config_pr::ChangedConfigPr { agent, rev, pr } = changed;
|
||||
let stale: Vec<hive_jobq::NodeId> = sched
|
||||
.graph()
|
||||
.nodes()
|
||||
.filter(|n| n.state == hive_jobq::State::Pending)
|
||||
.filter(|n| {
|
||||
matches!(&n.payload, SwarmNodeKind::BuildAgent { agent: a, pr: p, .. } if *a == agent && *p == pr)
|
||||
})
|
||||
.map(|n| n.id)
|
||||
.collect();
|
||||
let superseded = stale
|
||||
.into_iter()
|
||||
.filter(|&id| sched.cancel_node(id))
|
||||
.count();
|
||||
sched
|
||||
.insert_job(None, |b| {
|
||||
vec![
|
||||
b.node(SwarmNodeKind::BuildAgent { agent, rev, pr })
|
||||
.needs(SwarmResourceKind::Builder)
|
||||
.guid(),
|
||||
]
|
||||
})
|
||||
.map_err(|e| anyhow::anyhow!("{e}"))?;
|
||||
Ok(superseded)
|
||||
}
|
||||
|
||||
/// Queue the deploy of a merged config PR on the one hive that runs its agent.
|
||||
///
|
||||
/// The hive is found by scanning every hive's wanted state, since nothing
|
||||
|
|
@ -4147,6 +4182,101 @@ mod tests {
|
|||
}
|
||||
}
|
||||
|
||||
fn changed(agent: &str, rev: &str, pr: u64) -> crate::config_pr::ChangedConfigPr {
|
||||
crate::config_pr::ChangedConfigPr {
|
||||
agent: agent.to_owned(),
|
||||
rev: rev.to_owned(),
|
||||
pr,
|
||||
}
|
||||
}
|
||||
|
||||
fn build_state(
|
||||
sched: &hive_jobq::scheduler::Scheduler<SwarmNodeKind, super::SwarmResourceKind>,
|
||||
rev: &str,
|
||||
) -> hive_jobq::State {
|
||||
sched
|
||||
.graph()
|
||||
.nodes()
|
||||
.find(|n| matches!(&n.payload, SwarmNodeKind::BuildAgent { rev: r, .. } if r == rev))
|
||||
.map(|n| n.state)
|
||||
.expect("a build of that rev")
|
||||
}
|
||||
|
||||
/// A newer head of the same PR replaces its pending build; another PR's
|
||||
/// build, and the same PR's running build, are left alone.
|
||||
#[tokio::test]
|
||||
async fn a_newer_push_replaces_the_prs_pending_build_only() {
|
||||
let sched =
|
||||
std::sync::Arc::new(std::sync::Mutex::new(hive_jobq::scheduler::Scheduler::new(
|
||||
hive_jobq::Graph::new(),
|
||||
hive_jobq::resources::ResourceTable::new(),
|
||||
)));
|
||||
let queue = |c| super::queue_build(&mut sched.lock().unwrap(), c).expect("queued");
|
||||
assert_eq!(queue(changed("atlas", "r1", 3)), 0);
|
||||
// r1 takes the builder and stays running: its future is never awaited.
|
||||
let _running = hive_jobq::scheduler::Scheduler::claim_next(&sched, |_, _, b| async move {
|
||||
(b, hive_jobq::scheduler::Outcome::Done)
|
||||
})
|
||||
.expect("r1 is runnable");
|
||||
assert_eq!(
|
||||
queue(changed("atlas", "r2", 3)),
|
||||
0,
|
||||
"r1 is running, not pending"
|
||||
);
|
||||
assert_eq!(queue(changed("argus", "o1", 4)), 0);
|
||||
assert_eq!(queue(changed("atlas", "r3", 3)), 1, "r2 was pending");
|
||||
|
||||
let guard = sched.lock().unwrap();
|
||||
assert_eq!(build_state(&guard, "r1"), hive_jobq::State::Running);
|
||||
assert_eq!(build_state(&guard, "r2"), hive_jobq::State::Cancelled);
|
||||
assert_eq!(build_state(&guard, "r3"), hive_jobq::State::Pending);
|
||||
assert_eq!(build_state(&guard, "o1"), hive_jobq::State::Pending);
|
||||
}
|
||||
|
||||
/// Builds share one builder slot: a second build is not claimable while
|
||||
/// the first runs, and is once it has finished.
|
||||
#[tokio::test]
|
||||
async fn one_build_runs_at_a_time() {
|
||||
let sched =
|
||||
std::sync::Arc::new(std::sync::Mutex::new(hive_jobq::scheduler::Scheduler::new(
|
||||
hive_jobq::Graph::new(),
|
||||
hive_jobq::resources::ResourceTable::new(),
|
||||
)));
|
||||
for c in [changed("atlas", "a1", 1), changed("argus", "b1", 2)] {
|
||||
super::queue_build(&mut sched.lock().unwrap(), c).expect("queued");
|
||||
}
|
||||
let run = |sched| {
|
||||
hive_jobq::scheduler::Scheduler::claim_next(sched, |_, _, b| async move {
|
||||
(b, hive_jobq::scheduler::Outcome::Done)
|
||||
})
|
||||
};
|
||||
let first = run(&sched).expect("one build is runnable");
|
||||
assert!(
|
||||
run(&sched).is_none(),
|
||||
"a second build started beside the first"
|
||||
);
|
||||
first.await.1.expect("no growth");
|
||||
run(&sched)
|
||||
.expect("the second build runs once the first is done")
|
||||
.await
|
||||
.1
|
||||
.expect("no growth");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_new_agent_named_like_a_service_forge_account_is_refused() {
|
||||
let raw = nix_reserved();
|
||||
for name in [
|
||||
crate::forge::builder_account::LOGIN,
|
||||
crate::forge::classifier_account::LOGIN,
|
||||
] {
|
||||
let mut warnings = Vec::new();
|
||||
let broken = super::broken_name_rules(name, Some(&raw), &mut warnings);
|
||||
let err = super::name_verdict(name, &broken, Ok(false), &mut warnings).expect_err(name);
|
||||
assert!(err.contains("nix/reserved-names.nix"), "{name:?}: {err}");
|
||||
}
|
||||
}
|
||||
|
||||
/// One case per shape rule, each named in the refusal. Checked against a
|
||||
/// list without them, so the literal half cannot carry this test.
|
||||
#[test]
|
||||
|
|
|
|||
Loading…
Reference in a new issue