review classifier: the review table shows a pass/fail symbol, certainty against threshold, and an emoji on the overall line
The review body's table is now: result | check | certainty / threshold | reason. The result cell is a symbol for what the row did to the review, after polarity (✅ counted and passed, ❌ counted and failed, ❔ not counted or no answer), then the model's verdict. The overall line carries ✅ / ❌. Polarity, threshold and counted are folded into those cells. The prompt asks for a reason under 350 characters; the enforced cap stays 500, so an overshoot is truncated less often.
This commit is contained in:
parent
46fd6e502a
commit
731cef4c2d
5 changed files with 172 additions and 43 deletions
|
|
@ -12,6 +12,11 @@ use swarm_queue_client::review_classifier::Verdict;
|
|||
/// truncated to this, ending in `…`.
|
||||
pub const REASON_MAX_CHARS: usize = 500;
|
||||
|
||||
/// The length the prompt asks a reason to stay under. Below `REASON_MAX_CHARS`
|
||||
/// so a model that overshoots the ask still lands inside the cap, uncut.
|
||||
pub const REASON_ASKED_CHARS: usize = 350;
|
||||
const _: () = assert!(REASON_ASKED_CHARS < REASON_MAX_CHARS);
|
||||
|
||||
/// One validated answer.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct Answer {
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@ use std::fmt::Write as _;
|
|||
|
||||
use swarm_queue_client::review_classifier::{Classification, Polarity, TryOut};
|
||||
|
||||
use crate::answer::REASON_MAX_CHARS;
|
||||
use crate::answer::REASON_ASKED_CHARS;
|
||||
|
||||
/// The system prompt, the same for every run.
|
||||
pub const SYSTEM: &str = "\
|
||||
|
|
@ -89,7 +89,7 @@ pub fn classification_prompt<'a>(shared: &'a str, c: &Classification) -> Prompt<
|
|||
};
|
||||
let question = format!(
|
||||
"## The question\n\nClassification `{}` ({wanted}):\n\n{}\n\n\
|
||||
Answer with the JSON object only. Keep the reason under {REASON_MAX_CHARS} characters.\n",
|
||||
Answer with the JSON object only. Keep the reason under {REASON_ASKED_CHARS} characters.\n",
|
||||
c.name,
|
||||
c.description.trim()
|
||||
);
|
||||
|
|
@ -296,6 +296,16 @@ diff --git a/src/main.rs b/src/main.rs
|
|||
assert_eq!(shared, shared_context(&run()));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_question_asks_for_a_reason_under_the_asked_length_not_the_cap() {
|
||||
let c = classification("datetime", "d", Polarity::Positive);
|
||||
let question = classification_prompt("", &c).question;
|
||||
assert!(
|
||||
question.contains("Keep the reason under 350 characters."),
|
||||
"{question}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_shared_context_has_the_full_commit_message_and_a_collapsed_lockfile() {
|
||||
let shared = shared_context(&run());
|
||||
|
|
|
|||
Loading…
Reference in a new issue