vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-controller: record the review classifier's runs and list them

Browse source 
Every run is recorded when it ends: a try-out when its reply settles it,
a requested review once its review is posted or the post fails, and a
request turned away by a full queue once its comment is posted. A
duplicate request that starts nothing is not recorded. The newest 100
are one JSON array under `recent` in a new controller-owned KV bucket,
`review-classifier-runs`, written with compare-and-set as
`schedule-definitions` is (five attempts). A failed write is logged; the
review is already posted by then.

Each recorded run is also one info event, "review classifier run
finished", with the record's fields as tracing fields. The controller
logs natively to the journal, so they reach VictoriaLogs as F_<NAME>.

GET /api/review-classifier/runs returns the list, newest first, gated
by the gateway like the try-out GET. post_classifier_review now returns
the posted review's html_url, which the record keeps as review_url.

Closes #5153, part of #5096.
This commit is contained in:
atlas 2026-10-12 19:23:28 +02:00
commit 2fc0166427
14 changed files with 1138 additions and 66 deletions
Download patch file Download diff file Expand all files Collapse all files

576
swarm-controller/src/review_classifier/runs.rs Normal file
View file

@ -0,0 +1,576 @@
//! `GET /api/review-classifier/runs`: the review classifier's finished runs,
//! try-outs and requested reviews both, newest first.
//!
//! The newest [`MAX_RUNS`] are one JSON array under [`KEY`] in [`BUCKET`],
//! which only this daemon reads or writes. Recording a run is one
//! compare-and-set on that key, as for `schedules::DEFINITIONS_KEY`; one key,
//! because listing a bucket's keys needs a consumer grant this daemon holds
//! for no bucket of its own.
//!
//! Every recorded run is also one `info` event, [`EVENT`], carrying the
//! record's fields, so a run whose write failed is still in the log store.
//! A record holds no diff, no commit text and no token count or cost.
use std::future::Future;
use anyhow::Context as _;
use async_nats::jetstream::kv::Store;
use axum::Json;
use axum::extract::State;
use axum::http::StatusCode;
use problem_details::ProblemDetails;
use serde::{Deserialize, Serialize};
use swarm_queue_client::review_classifier::{Overall, Reply, Row};
use utoipa::ToSchema;
use crate::schedules::{read, write};
use crate::{AppState, error_problem};
/// The bucket holding the runs. Agents hold no grant on it.
pub const BUCKET: &str = "review-classifier-runs";
/// The one key in [`BUCKET`].
pub const KEY: &str = "recent";
/// How many runs the list keeps; recording one more drops the oldest.
pub(crate) const MAX_RUNS: usize = 100;
/// How many times a losing writer re-reads and re-applies, as in
/// `schedules::MAX_ATTEMPTS`.
const MAX_ATTEMPTS: usize = 5;
/// The message of the event [`Runs::record`] logs for every run.
pub(crate) const EVENT: &str = "review classifier run finished";
/// What started a run.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, ToSchema)]
#[serde(rename_all = "kebab-case")]
pub(crate) enum Trigger {
/// The swarm UI's try-out page.
TryOut,
/// A review request on the forge.
Review,
}
/// How a run ended.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, ToSchema)]
#[serde(rename_all = "kebab-case")]
pub(crate) enum RunOutcome {
/// A review approved the PR.
Approved,
/// A review requested changes.
ChangesRequested,
/// A comment was posted: the rules did not load, the PR was unreadable,
/// the run failed, the classifier sent a different number of answers
/// than it was sent classifications, or any classification got none.
NoVerdict,
/// No run started, as every run and wait place was taken; a comment
/// saying so was posted.
QueueFull,
/// The review could not be posted.
PostFailed,
/// A try-out the classifier answered.
Done,
/// A try-out that failed.
Failed,
}
/// One finished run.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, ToSchema)]
pub(crate) struct RunRecord {
/// A try-out's id is the one its POST answered with.
pub(super) id: String,
pub(super) trigger: Trigger,
/// `owner/name`.
pub(super) repo: String,
pub(super) pr: u64,
/// The swarm UI user who started a try-out, or the forge user who
/// requested the review.
pub(super) requester: String,
/// The commit the run read. Absent when the PR was not read.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub(super) head_sha: Option<String>,
/// Unix seconds: a try-out's POST, or a review's start once it had a
/// run place.
pub(super) started_unix: i64,
pub(super) finished_unix: i64,
pub(super) outcome: RunOutcome,
/// Why, for every outcome but `approved`, `changes-requested` and
/// `done`.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub(super) detail: Option<String>,
/// As the try-out GET returns them, when the classifier answered.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub(super) rows: Option<Vec<Row>>,
/// As the try-out GET returns it, when the classifier answered.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub(super) overall: Option<Overall>,
/// The posted review, when one was posted and the forge named its URL.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub(super) review_url: Option<String>,
}
/// The rows and the overall verdict of `reply`, when the classifier
/// answered.
pub(super) fn answer(reply: &Reply) -> (Option<Vec<Row>>, Option<Overall>) {
match reply {
Reply::Done { rows, overall } => (Some(rows.clone()), Some(*overall)),
Reply::Failed { .. } => (None, None),
}
}
/// Where the runs are kept: [`BUCKET`] in production, a fake in the tests.
pub(crate) trait RunDoc {
/// The runs, with the revision to write them back against.
fn read(&self) -> impl Future<Output = anyhow::Result<(Vec<RunRecord>, Option<u64>)>> + Send;
/// `Ok(false)` when another write landed since `revision`.
fn write(
&self,
runs: Vec<RunRecord>,
revision: Option<u64>,
) -> impl Future<Output = anyhow::Result<bool>> + Send;
}
impl RunDoc for Store {
async fn read(&self) -> anyhow::Result<(Vec<RunRecord>, Option<u64>)> {
read(self, KEY).await
}
async fn write(&self, runs: Vec<RunRecord>, revision: Option<u64>) -> anyhow::Result<bool> {
write(self, KEY, &runs, revision).await
}
}
/// Put `run` first in `doc`, keeping the newest [`MAX_RUNS`], re-reading and
/// re-applying when another write landed in between.
async fn push(doc: &impl RunDoc, run: &RunRecord) -> anyhow::Result<()> {
for _ in 0..MAX_ATTEMPTS {
let (runs, revision) = doc.read().await?;
if doc.write(newest(run, runs), revision).await? {
return Ok(());
}
tracing::debug!("review classifier runs: write lost a race, retrying");
}
anyhow::bail!("gave up recording the run after {MAX_ATTEMPTS} conflicting writes")
}
/// `runs` with `run` in front, cut to [`MAX_RUNS`].
fn newest(run: &RunRecord, mut runs: Vec<RunRecord>) -> Vec<RunRecord> {
runs.insert(0, run.clone());
runs.truncate(MAX_RUNS);
runs
}
/// The runs' store, as [`AppState`] holds it.
#[derive(Clone)]
pub(crate) enum Runs {
/// [`BUCKET`] on the swarm queue, when this host has one.
Live(Option<async_nats::Client>),
/// An in-memory document a test reads back.
#[cfg(test)]
Fake(std::sync::Arc<tests::FakeDoc>),
}
/// Why the runs' store could not be reached: always 503.
struct Unreachable(String);
impl Runs {
/// Log `run` and add it to the list. A failed write is logged and
/// changes nothing else.
pub(crate) async fn record(&self, run: RunRecord) {
log(&run);
if let Err(e) = self.push(&run).await {
tracing::warn!(
id = %run.id, error = %format!("{e:#}"),
"storing a review classifier run failed; the run is in this log only"
);
}
}
async fn push(&self, run: &RunRecord) -> anyhow::Result<()> {
match self {
Self::Live(client) => {
let store = open(client.as_ref()).await.map_err(|Unreachable(e)| {
anyhow::anyhow!(e).context(format!("opening the {BUCKET} bucket"))
})?;
push(&store, run).await
}
#[cfg(test)]
Self::Fake(doc) => push(&**doc, run).await,
}
}
async fn list(&self) -> Result<Vec<RunRecord>, ProblemDetails> {
let read = match self {
Self::Live(client) => {
let store = open(client.as_ref())
.await
.map_err(|Unreachable(e)| error_problem(StatusCode::SERVICE_UNAVAILABLE, &e))?;
RunDoc::read(&store).await
}
#[cfg(test)]
Self::Fake(doc) => doc.read().await,
};
read.map(|(runs, _)| runs).map_err(|e| {
let detail = format!("{e:#}");
tracing::warn!(error = %detail, "reading the review classifier runs failed");
error_problem(StatusCode::INTERNAL_SERVER_ERROR, &detail)
})
}
}
/// Open [`BUCKET`], creating it if nothing has yet.
async fn open(client: Option<&async_nats::Client>) -> Result<Store, Unreachable> {
let client = client
.ok_or_else(|| Unreachable("no swarm queue is configured on this host".to_owned()))?;
swarm_queue_client::ensure_connected(client)
.map_err(|e| Unreachable(swarm_queue_client::chain(&e)))?;
let js = async_nats::jetstream::new(client.clone());
if let Ok(store) = js.get_key_value(BUCKET).await {
return Ok(store);
}
js.create_key_value(async_nats::jetstream::kv::Config {
bucket: BUCKET.to_owned(),
description: "The review classifier's newest finished runs".to_owned(),
history: 1,
..Default::default()
})
.await
.with_context(|| format!("creating the {BUCKET} bucket"))
.map_err(|e| Unreachable(format!("{e:#}")))
}
/// `value` as its one serde word, as the route spells it.
fn word<T: Serialize>(value: &T) -> String {
serde_json::to_value(value)
.ok()
.and_then(|v| v.as_str().map(str::to_owned))
.unwrap_or_default()
}
/// The [`EVENT`] for `run`, one field per record field. `rows` is the JSON
/// the route returns for them.
fn log(run: &RunRecord) {
let rows = run
.rows
.as_ref()
.and_then(|rows| serde_json::to_string(rows).ok());
tracing::info!(
id = %run.id,
trigger = %word(&run.trigger),
repo = %run.repo,
pr = run.pr,
requester = %run.requester,
head_sha = run.head_sha.as_deref(),
started_unix = run.started_unix,
finished_unix = run.finished_unix,
outcome = %word(&run.outcome),
detail = run.detail.as_deref(),
rows = rows.as_deref(),
overall = run.overall.as_ref().map(word).as_deref(),
review_url = run.review_url.as_deref(),
"{EVENT}"
);
}
/// List the review classifier's finished runs.
#[utoipa::path(
get,
path = "/api/review-classifier/runs",
responses(
(status = 200, description = "the newest 100 finished runs, try-outs and requested \
reviews, newest first; empty when there are none", body = Vec<RunRecord>),
(status = 503, description = "no swarm queue is configured on this host, it is not \
connected, or the runs' bucket could not be opened (problem+json)", body = String),
(status = 500, description = "the runs could not be read (problem+json)",
body = String),
),
tag = "review-classifier"
)]
pub(crate) async fn get_runs(
State(state): State<AppState>,
) -> Result<Json<Vec<RunRecord>>, ProblemDetails> {
state.runs.list().await.map(Json)
}
#[cfg(test)]
pub(crate) mod tests {
use std::sync::{Arc, Mutex};
use axum::body::Body;
use axum::http::Request;
use swarm_queue_client::review_classifier::{Polarity, Verdict};
use tower::ServiceExt as _;
use tracing_subscriber::layer::SubscriberExt as _;
use super::*;
/// A document in memory. Each of the first `conflicts` writes loses to a
/// write that lands one run in between; a document set to `fail` fails
/// every read.
#[derive(Default)]
pub(crate) struct FakeDoc {
state: Mutex<(Vec<RunRecord>, u64)>,
conflicts: Mutex<usize>,
fail: bool,
}
impl FakeDoc {
pub(crate) fn failing() -> Self {
Self {
fail: true,
..Self::default()
}
}
fn conflicting(conflicts: usize) -> Self {
Self {
conflicts: Mutex::new(conflicts),
..Self::default()
}
}
pub(crate) fn runs(&self) -> Vec<RunRecord> {
self.state.lock().expect("unpoisoned").0.clone()
}
}
impl RunDoc for FakeDoc {
async fn read(&self) -> anyhow::Result<(Vec<RunRecord>, Option<u64>)> {
anyhow::ensure!(!self.fail, "the store is down");
let (runs, revision) = self.state.lock().expect("unpoisoned").clone();
Ok((runs, Some(revision).filter(|r| *r > 0)))
}
async fn write(&self, runs: Vec<RunRecord>, revision: Option<u64>) -> anyhow::Result<bool> {
let mut state = self.state.lock().expect("unpoisoned");
let mut conflicts = self.conflicts.lock().expect("unpoisoned");
if *conflicts > 0 {
*conflicts -= 1;
let mut other = record(&format!("other-{conflicts}"));
other.trigger = Trigger::Review;
state.0.insert(0, other);
state.1 += 1;
}
if revision.unwrap_or(0) != state.1 {
return Ok(false);
}
*state = (runs, state.1 + 1);
Ok(true)
}
}
pub(crate) fn record(id: &str) -> RunRecord {
RunRecord {
id: id.to_owned(),
trigger: Trigger::TryOut,
repo: "o/r".to_owned(),
pr: 7,
requester: "alice".to_owned(),
head_sha: Some("abc123".to_owned()),
started_unix: 1_000,
finished_unix: 1_060,
outcome: RunOutcome::Done,
detail: None,
rows: None,
overall: None,
review_url: None,
}
}
fn ids(runs: &[RunRecord]) -> Vec<&str> {
runs.iter().map(|r| r.id.as_str()).collect()
}
#[tokio::test]
async fn the_newest_hundred_survive_newest_first() {
let doc = FakeDoc::default();
for i in 0..=MAX_RUNS {
push(&doc, &record(&format!("r{i}"))).await.expect("stored");
}
let runs = doc.runs();
assert_eq!(runs.len(), 100);
assert_eq!(runs[0].id, "r100");
assert_eq!(runs[99].id, "r1", "r0, the oldest, is gone");
}
#[tokio::test]
async fn a_write_that_lost_a_race_is_redone_on_the_winner() {
let doc = FakeDoc::conflicting(2);
push(&doc, &record("mine")).await.expect("stored");
assert_eq!(ids(&doc.runs()), ["mine", "other-0", "other-1"]);
let doc = FakeDoc::conflicting(MAX_ATTEMPTS);
let err = push(&doc, &record("mine")).await.expect_err("gives up");
assert!(err.to_string().contains("conflicting writes"), "{err}");
assert!(!ids(&doc.runs()).contains(&"mine"));
}
/// The bucket on a real server: created on first use, written, read back.
#[tokio::test]
async fn runs_round_trip_through_the_bucket() {
let nats = crate::test_nats::TestNats::start();
let runs = Runs::Live(Some(nats.client().await));
runs.record(record("a")).await;
runs.record(record("b")).await;
let listed = runs.list().await.expect("listed");
assert_eq!(ids(&listed), ["b", "a"]);
}
#[tokio::test]
async fn the_route_lists_the_runs_newest_first_and_a_down_store_is_an_error() {
let doc = Arc::new(FakeDoc::default());
let mut reviewed = record("b");
reviewed.trigger = Trigger::Review;
reviewed.outcome = RunOutcome::NoVerdict;
reviewed.detail = Some("the rules did not load".to_owned());
reviewed.head_sha = None;
push(&*doc, &record("a")).await.expect("stored");
push(&*doc, &reviewed).await.expect("stored");
let get = |runs: Runs| async move {
let (mut state, _) = crate::tests::state_with_roster();
state.runs = runs;
let res = crate::build_app(state)
.oneshot(
Request::get("/api/review-classifier/runs")
.body(Body::empty())
.expect("a request"),
)
.await
.expect("infallible");
let status = res.status();
let bytes = axum::body::to_bytes(res.into_body(), usize::MAX)
.await
.expect("a body");
let body: serde_json::Value = serde_json::from_slice(&bytes).expect("JSON");
(status, body)
};
let (status, body) = get(Runs::Fake(doc)).await;
assert_eq!(status, StatusCode::OK, "{body}");
assert_eq!(body[0]["id"], "b");
assert_eq!(body[0]["trigger"], "review");
assert_eq!(body[0]["outcome"], "no-verdict");
assert_eq!(body[0]["detail"], "the rules did not load");
assert!(body[0].get("head_sha").is_none(), "{body}");
assert_eq!(body[1]["trigger"], "try-out");
assert_eq!(body[1]["outcome"], "done");
let (status, body) = get(Runs::Fake(Arc::new(FakeDoc::failing()))).await;
assert_eq!(status, StatusCode::INTERNAL_SERVER_ERROR, "{body}");
let (status, body) = get(Runs::Live(None)).await;
assert_eq!(status, StatusCode::SERVICE_UNAVAILABLE, "{body}");
assert_eq!(body["detail"], "no swarm queue is configured on this host");
}
#[test]
fn a_record_carries_no_diff_commit_text_or_cost() {
let mut run = record("a");
run.rows = Some(Vec::new());
run.overall = Some(Overall::Approve);
run.detail = Some("why".to_owned());
run.review_url = Some("https://forge/o/r/pulls/7#issuecomment-1".to_owned());
let value = serde_json::to_value(&run).expect("encodes");
let mut keys: Vec<_> = value.as_object().expect("an object").keys().collect();
keys.sort();
assert_eq!(
keys,
[
"detail",
"finished_unix",
"head_sha",
"id",
"outcome",
"overall",
"pr",
"repo",
"requester",
"review_url",
"rows",
"started_unix",
"trigger",
]
);
}
/// Every field of every event, by name.
#[derive(Clone, Default)]
struct Fields(Arc<Mutex<Vec<(String, String)>>>);
impl tracing::field::Visit for Fields {
fn record_debug(&mut self, field: &tracing::field::Field, value: &dyn std::fmt::Debug) {
self.0
.lock()
.expect("unpoisoned")
.push((field.name().to_owned(), format!("{value:?}")));
}
fn record_str(&mut self, field: &tracing::field::Field, value: &str) {
self.0
.lock()
.expect("unpoisoned")
.push((field.name().to_owned(), value.to_owned()));
}
}
impl<S: tracing::Subscriber> tracing_subscriber::Layer<S> for Fields {
fn on_event(
&self,
event: &tracing::Event<'_>,
_: tracing_subscriber::layer::Context<'_, S>,
) {
event.record(&mut self.clone());
}
}
#[test]
fn the_log_event_carries_the_records_fields() {
let mut run = record("a");
run.trigger = Trigger::Review;
run.outcome = RunOutcome::ChangesRequested;
run.rows = Some(vec![Row {
name: "datetime".to_owned(),
polarity: Polarity::Positive,
verdict: Verdict::NoMatch,
certainty: 90,
threshold: 80,
counted: true,
passed: false,
reason: "No zone.".to_owned(),
error: None,
}]);
run.overall = Some(Overall::RequestChanges);
run.review_url = Some("https://forge/o/r/pulls/7#issuecomment-1".to_owned());
let fields = Fields::default();
let subscriber = tracing_subscriber::registry().with(fields.clone());
tracing::subscriber::with_default(subscriber, || log(&run));
let got = fields.0.lock().expect("unpoisoned").clone();
let field = |name: &str| {
got.iter()
.find(|(n, _)| n == name)
.map_or_else(|| panic!("no {name} in {got:?}"), |(_, v)| v.clone())
};
assert_eq!(field("message"), EVENT);
assert_eq!(field("id"), "a");
assert_eq!(field("trigger"), "review");
assert_eq!(field("repo"), "o/r");
assert_eq!(field("pr"), "7");
assert_eq!(field("requester"), "alice");
assert_eq!(field("head_sha"), "abc123");
assert_eq!(field("started_unix"), "1000");
assert_eq!(field("finished_unix"), "1060");
assert_eq!(field("outcome"), "changes-requested");
assert_eq!(field("overall"), "request-changes");
assert_eq!(
field("review_url"),
"https://forge/o/r/pulls/7#issuecomment-1"
);
let rows: Vec<Row> = serde_json::from_str(&field("rows")).expect("the rows as JSON");
assert_eq!(Some(rows), run.rows);
assert!(
!got.iter().any(|(n, _)| n == "detail"),
"an absent field is left out: {got:?}"
);
}
}