review-classifier: ask a check once more when its answer isn't valid JSON
An answer that fails to validate is asked for a second time, for that classification only, with a reminder to reply with the JSON object alone. The second ask comes out of the classification's CLASSIFICATION_DEADLINE: it gets what the first ask left, and is skipped when less than 20 seconds remain, so RUN_DEADLINE's bound still holds. A failed ask (runtime error, timeout) is not repeated. If the second answer does not validate either, the row has no answer, as before. Model::ask takes the deadline for the ask; the ACP runtime no longer holds its own. The parser also takes the first fenced block of a reply that has prose around the fence; prose around a bare object is still rejected.
This commit is contained in:
parent
b8897e7c7a
commit
ce7b3e055f
8 changed files with 345 additions and 66 deletions
|
|
@ -33,8 +33,8 @@ struct Raw {
|
|||
reason: String,
|
||||
}
|
||||
|
||||
/// Parse `text`, the model's final message. A surrounding code fence is
|
||||
/// tolerated; anything else around the object is not.
|
||||
/// Parse `text`, the model's final message. A code fence around the object is
|
||||
/// tolerated, with prose around the fence; prose around a bare object is not.
|
||||
///
|
||||
/// # Errors
|
||||
/// A message describing what is wrong with the answer, without quoting it.
|
||||
|
|
@ -80,10 +80,19 @@ fn cap_reason(reason: String) -> String {
|
|||
cut
|
||||
}
|
||||
|
||||
/// `text` without a surrounding code fence, with or without a language tag.
|
||||
/// `text` without its code fence, with or without a language tag. A fence
|
||||
/// around the whole message is stripped; otherwise the first fenced block in
|
||||
/// the message is taken, so prose before or after it is dropped.
|
||||
fn unfence(text: &str) -> &str {
|
||||
let Some(inner) = text.strip_prefix("```").and_then(|t| t.strip_suffix("```")) else {
|
||||
return text;
|
||||
let inner = match text.strip_prefix("```").and_then(|t| t.strip_suffix("```")) {
|
||||
Some(inner) => inner,
|
||||
None => match text
|
||||
.split_once("```")
|
||||
.and_then(|(_, t)| t.split_once("```"))
|
||||
{
|
||||
Some((block, _)) => block,
|
||||
None => return text,
|
||||
},
|
||||
};
|
||||
let inner = inner.strip_prefix("json").unwrap_or(inner);
|
||||
inner.trim()
|
||||
|
|
@ -113,6 +122,20 @@ mod tests {
|
|||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_fenced_answer_inside_prose_parses() {
|
||||
let a = parse(
|
||||
"Here is my answer:\n```json\n{\"verdict\":\"match\",\"certainty\":80,\"reason\":\"x\"}\n```\nHope that helps.",
|
||||
)
|
||||
.expect("fence in prose");
|
||||
assert_eq!((a.verdict, a.certainty), (Verdict::Match, 80));
|
||||
let untagged =
|
||||
"Answer: ```{\"verdict\":\"no-match\",\"certainty\":70,\"reason\":\"\"}``` done";
|
||||
assert_eq!(parse(untagged).expect("untagged").verdict, Verdict::NoMatch);
|
||||
assert!(parse("Here is my answer:\n```json\nnot json\n```").is_err());
|
||||
assert!(parse("an unclosed ```json\n{\"verdict\":\"match\"").is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_malformed_answer_is_an_error() {
|
||||
for bad in [
|
||||
|
|
|
|||
Loading…
Reference in a new issue