vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

review classifier: model, effort and review concurrency as swarm-wide settings

Browse source 
The classifier ran every classification on a hard-coded `haiku` with no
`--effort`, and the controller ran at most two requested reviews at once,
also hard-coded, so changing any of them took a code change and a
redeploy.

swarm-controller now keeps one settings document,
`{model, effort, max_concurrent_reviews}`, in its own
`review-classifier-settings` KV bucket under `all`, the way it keeps the
schedule definitions. No document means today's values: haiku, no
`--effort`, two reviews. `GET /api/review-classifier/settings` reads it;
`PUT` replaces it whole, needs the gateway's `Remote-User` like the
try-out POST, and refuses a bad model (the inference route's check), an
effort outside the five levels or a limit outside 1..=8 with
problem+json 400.

Every run request (try-out and requested review) now carries the model
and effort, and the classifier passes them to claude instead of its
constant, which is gone. A try-out reads the settings when it is posted;
a review reads them for the limit before it waits for a place and again
for the model and effort once it has one. The review semaphore is
replaced by a running count under a resizable limit, so a PUT, or the
next review's read, changes how many run without a restart, and a lower
limit lets the runs already going finish. MAX_WAITING_REVIEWS is
unchanged.

Closes #5151, part of #5096.
This commit is contained in:
atlas 2026-10-12 19:21:33 +02:00 • committed by mara
commit 2c5eef73cd
10 changed files with 868 additions and 69 deletions
Download patch file Download diff file Expand all files Collapse all files

312
swarm-controller/src/review_classifier/review.rs
View file

@ -8,29 +8,28 @@
//! official reviews count toward approvals or block a merge.
//!
//! A run with no clean verdict never approves: see [`no_clean_verdict_event`].
//! At most [`MAX_CONCURRENT_REVIEWS`] runs go at once and at most
//! At most `max_concurrent_reviews` runs go at once, as the swarm's
//! [settings](super::settings) say when a run asks for its place, and at most
//! [`MAX_WAITING_REVIEWS`] more wait here, in memory, so a restart drops them.
//! See [`Reviews`] for what happens to a request beyond that.
use std::collections::HashSet;
use std::fmt::Write as _;
use std::future::Future;
use std::sync::{Arc, Mutex, PoisonError};
use std::sync::{Arc, Mutex, MutexGuard, PoisonError};
use serde::Deserialize;
use swarm_queue_client::review_classifier::{Overall, Reply, Row, TryOut, Verdict};
use tokio::sync::Semaphore;
use tokio::sync::Notify;
use tokio::task::JoinHandle;
use super::runs::{self, RunOutcome, RunRecord, Runs, Trigger};
use super::settings::{Settings, SettingsSource};
use super::{FileSource, PullSource, RunSink, split_repo};
use crate::AppState;
use crate::agent_renewal::unix_now;
use crate::forge::{self, classifier_account::LOGIN, classifier_account::Review};
/// The most review runs in flight at once.
const MAX_CONCURRENT_REVIEWS: usize = 2;
/// The most review runs waiting for a slot.
const MAX_WAITING_REVIEWS: usize = 8;
@ -144,17 +143,29 @@ type Key = (String, String, i64, String);
/// The review runs this controller has accepted and not yet finished.
///
/// A request whose [`Key`] is already running or waiting starts nothing. A
/// request beyond [`MAX_CONCURRENT_REVIEWS`] + [`MAX_WAITING_REVIEWS`] starts
/// no run either; the account posts a no-verdict comment saying so.
/// request beyond the run limit + [`MAX_WAITING_REVIEWS`] starts no run
/// either; the account posts a no-verdict comment saying so.
pub(crate) struct Reviews {
slots: Arc<Semaphore>,
slots: Mutex<Slots>,
/// Woken when a run ends or the limit changes.
changed: Notify,
accepted: Mutex<HashSet<Key>>,
}
/// How many runs may go at once, and how many do.
struct Slots {
limit: usize,
running: usize,
}
impl Default for Reviews {
fn default() -> Self {
Self {
slots: Arc::new(Semaphore::new(MAX_CONCURRENT_REVIEWS)),
slots: Mutex::new(Slots {
limit: Settings::default().max_concurrent_reviews,
running: 0,
}),
changed: Notify::new(),
accepted: Mutex::default(),
}
}
@ -165,18 +176,34 @@ impl Default for Reviews {
enum Refused {
/// A run for the same key is running or waiting.
Duplicate,
/// Every run and wait place is taken.
Full,
/// Every run and wait place is taken, by this many runs.
Full(usize),
}
impl Reviews {
fn slots(&self) -> MutexGuard<'_, Slots> {
self.slots.lock().unwrap_or_else(PoisonError::into_inner)
}
/// Let `limit` runs go at once from now on. Runs already going when it
/// shrinks finish; no new one starts until fewer than `limit` run.
pub(crate) fn resize(&self, limit: usize) {
self.slots().limit = limit;
self.changed.notify_waiters();
}
#[cfg(test)]
pub(crate) fn limit(&self) -> usize {
self.slots().limit
}
fn accept(self: &Arc<Self>, key: Key) -> Result<Accepted, Refused> {
let mut accepted = self.accepted.lock().unwrap_or_else(PoisonError::into_inner);
if accepted.contains(&key) {
return Err(Refused::Duplicate);
}
if accepted.len() >= MAX_CONCURRENT_REVIEWS + MAX_WAITING_REVIEWS {
return Err(Refused::Full);
if accepted.len() >= self.slots().limit + MAX_WAITING_REVIEWS {
return Err(Refused::Full(accepted.len()));
}
accepted.insert(key.clone());
Ok(Accepted {
@ -184,6 +211,23 @@ impl Reviews {
key,
})
}
/// A place to run in, once fewer runs go than the limit allows.
async fn slot(self: &Arc<Self>) -> Slot {
loop {
let changed = self.changed.notified();
tokio::pin!(changed);
changed.as_mut().enable();
{
let mut slots = self.slots();
if slots.running < slots.limit {
slots.running += 1;
return Slot(Arc::clone(self));
}
}
changed.await;
}
}
}
/// One accepted run; dropping it frees its key and its place.
@ -202,6 +246,16 @@ impl Drop for Accepted {
}
}
/// One run holding a place; dropping it lets the next one start.
struct Slot(Arc<Reviews>);
impl Drop for Slot {
fn drop(&mut self) {
self.0.slots().running -= 1;
self.0.changed.notify_waiters();
}
}
/// Start a review run for a verified `review-request` delivery that asks for
/// [`LOGIN`], and return at once.
pub(crate) fn on_delivery(state: &AppState, body: &[u8]) {
@ -231,6 +285,9 @@ pub(crate) fn on_delivery(state: &AppState, body: &[u8]) {
/// Accept `request` into `reviews` and spawn its run, or the no-verdict
/// comment when there's no place for it, and record it in `runs` when it
/// ends. The task, if one was spawned.
///
/// The run reads the settings for the limit before it waits for a place, and
/// again for the model and effort once it has one.
fn start<S, Q, P>(
reviews: &Arc<Reviews>,
forge: Arc<S>,
@ -241,18 +298,25 @@ fn start<S, Q, P>(
) -> Option<JoinHandle<()>>
where
S: FileSource + PullSource + Send + Sync + 'static,
Q: RunSink,
Q: RunSink + SettingsSource,
P: ReviewPoster + Send + Sync + 'static,
{
let repo = format!("{}/{}", request.owner, request.name);
match reviews.accept(request.key()) {
Ok(accepted) => {
let slots = Arc::clone(&reviews.slots);
let reviews = Arc::clone(reviews);
Some(tokio::spawn(async move {
let Ok(_slot) = slots.acquire_owned().await else {
return;
};
match queue.settings().await {
Ok(settings) => reviews.resize(settings.max_concurrent_reviews),
Err(e) => tracing::warn!(
error = %format!("{e:#}"),
"reading the review classifier's settings failed; the run waits under \
the last limit read"
),
}
let slot = reviews.slot().await;
let run = Box::pin(run(&*forge, &*forge, &queue, &*poster, &request)).await;
drop(slot);
drop(accepted);
runs.record(run).await;
}))
@ -264,14 +328,14 @@ where
);
None
}
Err(Refused::Full) => {
Err(Refused::Full(taken)) => {
tracing::warn!(
%repo, pr = request.pr, head_sha = request.head.as_str(),
"review requested, but every run and wait place is taken; posting a no-verdict comment"
);
Some(tokio::spawn(async move {
let started_unix = unix_now();
let ruling = queue_full_ruling();
let ruling = queue_full_ruling(taken);
let reason = ruling.reason.clone();
let sent = post(&*poster, &request, None, ruling.review()).await;
let ended = ended(RunOutcome::QueueFull, reason, sent);
@ -282,11 +346,12 @@ where
}
}
/// The ruling for a request [`Reviews`] had no place for.
fn queue_full_ruling() -> Ruling {
/// The ruling for a request [`Reviews`] had no place for, with `taken` runs
/// running or waiting.
fn queue_full_ruling(taken: usize) -> Ruling {
let reason = format!(
"{MAX_CONCURRENT_REVIEWS} reviews are running and {MAX_WAITING_REVIEWS} are waiting, so \
this request starts no run; request the review again later"
"{taken} reviews are running or waiting, so this request starts no run; \
request the review again later"
);
Ruling {
event: no_clean_verdict_event(),
@ -342,6 +407,16 @@ impl RunSink for Queue {
}
}
impl SettingsSource for Queue {
async fn settings(&self) -> anyhow::Result<Settings> {
let Some(client) = &self.0 else {
anyhow::bail!("no swarm queue is configured on this host");
};
swarm_queue_client::ensure_connected(client)?;
client.settings().await
}
}
/// How far a run got.
#[derive(Debug, Clone, PartialEq, Eq)]
enum Outcome {
@ -356,7 +431,7 @@ enum Outcome {
async fn run(
files: &impl FileSource,
pulls: &impl PullSource,
sink: &impl RunSink,
sink: &(impl RunSink + SettingsSource),
poster: &impl ReviewPoster,
req: &Requested,
) -> RunRecord {
@ -479,7 +554,7 @@ fn record(
async fn classify(
files: &impl FileSource,
pulls: &impl PullSource,
sink: &impl RunSink,
sink: &(impl RunSink + SettingsSource),
req: &Requested,
) -> (Option<String>, Outcome) {
let repo = format!("{}/{}", req.owner, req.name);
@ -514,6 +589,15 @@ async fn classify(
);
}
};
let settings = match sink.settings().await {
Ok(settings) => settings,
Err(e) => {
return (
head,
Outcome::NotRun(format!("reading the classifier's settings failed: {e:#}")),
);
}
};
let request = TryOut {
operator: req.requester.clone(),
repo,
@ -522,6 +606,8 @@ async fn classify(
commits: pull.commits,
diff: pull.diff,
classifications: rules.classifications,
model: settings.model,
effort: settings.effort,
};
let sent = request.classifications.len();
let outcome = match serde_json::to_vec(&request) {
@ -686,6 +772,7 @@ fn cell(text: &str) -> String {
#[cfg(test)]
mod tests {
use std::sync::Mutex;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::time::Duration;
use swarm_queue_client::review_classifier::Polarity;
@ -838,19 +925,44 @@ mod tests {
assert_eq!(ruling.event, Event::Comment);
}
/// A classifier that answers approve once [`Gate::open`] lets it through.
/// A classifier that counts the runs reaching it and answers approve once
/// [`Gate::open`] lets each through, beside a store holding `settings`.
#[derive(Clone)]
struct Gate(Arc<tokio::sync::Semaphore>);
struct Gate {
permits: Arc<tokio::sync::Semaphore>,
reached: Arc<AtomicUsize>,
settings: Arc<Mutex<Settings>>,
}
impl Gate {
fn open(&self) {
self.0.add_permits(1);
fn closed() -> Self {
Self {
permits: Arc::new(tokio::sync::Semaphore::new(0)),
reached: Arc::default(),
settings: Arc::default(),
}
}
fn open(&self, runs: usize) {
self.permits.add_permits(runs);
}
fn reached(&self) -> usize {
self.reached.load(Ordering::SeqCst)
}
fn store(&self, max_concurrent_reviews: usize) {
self.settings
.lock()
.expect("unpoisoned")
.max_concurrent_reviews = max_concurrent_reviews;
}
}
impl RunSink for Gate {
async fn run(&self, _request: Vec<u8>) -> Reply {
let _permit = self.0.acquire().await.expect("never closed");
self.reached.fetch_add(1, Ordering::SeqCst);
let _permit = self.permits.acquire().await.expect("never closed");
Reply::Done {
rows: vec![row(None, true, true)],
overall: Overall::Approve,
@ -858,6 +970,19 @@ mod tests {
}
}
impl SettingsSource for Gate {
async fn settings(&self) -> anyhow::Result<Settings> {
Ok(self.settings.lock().expect("unpoisoned").clone())
}
}
/// Let every spawned task run until it waits on something.
async fn settle() {
for _ in 0..200 {
tokio::task::yield_now().await;
}
}
fn requested_at(head: &str) -> Requested {
Requested {
head: head.to_owned(),
@ -874,7 +999,7 @@ mod tests {
},
vec![rules_json()],
));
let gate = Gate(Arc::new(tokio::sync::Semaphore::new(0)));
let gate = Gate::closed();
let poster = Arc::new(Posted::default());
let doc = Arc::new(FakeDoc::default());
let store = Runs::Fake(Arc::clone(&doc));
@ -896,7 +1021,7 @@ mod tests {
for head in ["a", "b", "c"] {
assert!(start_at(head).is_none(), "{head}");
}
gate.open();
gate.open(1);
for task in tasks.drain(..) {
task.await.expect("ran");
}
@ -922,7 +1047,7 @@ mod tests {
},
vec![rules_json()],
));
let gate = Gate(Arc::new(tokio::sync::Semaphore::new(0)));
let gate = Gate::closed();
let poster = Arc::new(Posted::default());
let doc = Arc::new(FakeDoc::default());
let store = Runs::Fake(Arc::clone(&doc));
@ -938,7 +1063,7 @@ mod tests {
.expect("a run or a comment")
};
let places = MAX_CONCURRENT_REVIEWS + MAX_WAITING_REVIEWS;
let places = Settings::default().max_concurrent_reviews + MAX_WAITING_REVIEWS;
let runs: Vec<_> = (0..places).map(|i| start_at(format!("h{i}"))).collect();
tokio::time::timeout(Duration::from_secs(5), start_at("over".to_owned()))
.await
@ -952,9 +1077,9 @@ mod tests {
assert_eq!(*commit, None);
assert_eq!(review.event, "COMMENT");
assert!(
review
.body
.starts_with("**No verdict:** 2 reviews are running and 8 are waiting"),
review.body.starts_with(
"**No verdict:** 10 reviews are running or waiting, so this request"
),
"{}",
review.body
);
@ -971,12 +1096,12 @@ mod tests {
refused[0]
.detail
.as_deref()
.is_some_and(|d| d.starts_with("2 reviews are running and 8 are waiting")),
.is_some_and(|d| d.starts_with("10 reviews are running or waiting")),
"{:?}",
refused[0].detail
);
gate.open();
gate.open(1);
for run in runs {
run.await.expect("ran");
}
@ -1166,6 +1291,111 @@ mod tests {
assert_eq!(all[0].2.event, "APPROVED");
}
#[tokio::test]
async fn a_new_limit_applies_to_the_next_run_without_a_restart() {
let reviews = Arc::new(Reviews::default());
let fake = Arc::new(Fake::new(
Reply::Failed {
error: String::new(),
},
vec![rules_json()],
));
let gate = Gate::closed();
let poster = Arc::new(Posted::default());
let start_at = |head: &str| {
start(
&reviews,
Arc::clone(&fake),
gate.clone(),
Arc::clone(&poster),
Runs::Fake(Arc::default()),
requested_at(head),
)
.expect("accepted")
};
// The stored limit is read before a run waits, not the default 2.
gate.store(1);
let mut tasks = vec![start_at("a"), start_at("b")];
settle().await;
assert_eq!(gate.reached(), 1);
// Raised, as a PUT does: the waiting run starts at once.
gate.store(3);
reviews.resize(3);
settle().await;
assert_eq!(gate.reached(), 2);
tasks.push(start_at("c"));
settle().await;
assert_eq!(gate.reached(), 3);
// Lowered: the three keep running and the next one waits.
gate.store(1);
tasks.push(start_at("d"));
settle().await;
assert_eq!(gate.reached(), 3);
gate.open(1);
for task in tasks {
tokio::time::timeout(Duration::from_secs(5), task)
.await
.expect("a finished run frees its place")
.expect("ran");
}
assert_eq!(gate.reached(), 4);
assert_eq!(reviews.slots().running, 0, "every place is freed");
assert_eq!(poster.0.lock().expect("unpoisoned").len(), 4);
}
#[tokio::test]
async fn a_review_runs_on_the_stored_model_and_effort() {
use swarm_queue_client::agent_inference::Effort;
let mut fake = Fake::new(
Reply::Done {
rows: vec![row(None, true, true)],
overall: Overall::Approve,
},
vec![rules_json()],
);
fake.settings = Settings {
model: "sonnet".to_owned(),
effort: Some(Effort::Medium),
max_concurrent_reviews: 2,
};
let poster = Posted::default();
run(&fake, &fake, &fake, &poster, &request("r")).await;
let sent = fake.sent.lock().expect("unpoisoned");
assert_eq!(sent.len(), 1);
assert_eq!(
(sent[0].model.as_str(), sent[0].effort),
("sonnet", Some(Effort::Medium))
);
}
#[tokio::test]
async fn unreadable_settings_run_nothing_and_post_a_comment() {
let fake = Fake::new(
Reply::Done {
rows: vec![row(None, true, true)],
overall: Overall::Approve,
},
vec![rules_json()],
);
let poster = Posted::default();
run(&fake, &fake, &Queue(None), &poster, &request("r")).await;
let (_, commit, review) = poster.0.into_inner().expect("unpoisoned").remove(0);
assert_eq!(review.event, "COMMENT");
assert!(
review
.body
.contains("reading the classifier's settings failed: no swarm queue is configured"),
"{}",
review.body
);
assert_eq!(commit.as_deref(), Some("abc123"));
}
#[tokio::test]
async fn a_clean_run_posts_its_verdict_on_the_head_commit() {
let (target, commit, review) = posted(