Watch
0
0
Fork
You've already forked hyperhive
0

review-classifier: ask a check once more when its answer isn't valid JSON

An answer that fails to validate is asked for a second time, for that
classification only, with a reminder to reply with the JSON object alone.
The second ask comes out of the classification's CLASSIFICATION_DEADLINE:
it gets what the first ask left, and is skipped when less than 20 seconds
remain, so RUN_DEADLINE's bound still holds. A failed ask (runtime error,
timeout) is not repeated. If the second answer does not validate either,
the row has no answer, as before.

Model::ask takes the deadline for the ask; the ACP runtime no longer holds
its own.

The parser also takes the first fenced block of a reply that has prose
around the fence; prose around a bare object is still rejected.
This commit is contained in:
atlas 2026-10-16 15:32:07 +02:00 • committed by mara
commit ce7b3e055f
8 changed files with 345 additions and 66 deletions

View file

@ -33,8 +33,8 @@ struct Raw {
reason: String,
}
/// Parse `text`, the model's final message. A surrounding code fence is
/// tolerated; anything else around the object is not.
/// Parse `text`, the model's final message. A code fence around the object is
/// tolerated, with prose around the fence; prose around a bare object is not.
///
/// # Errors
/// A message describing what is wrong with the answer, without quoting it.
@ -80,10 +80,19 @@ fn cap_reason(reason: String) -> String {
cut
}
/// `text` without a surrounding code fence, with or without a language tag.
/// `text` without its code fence, with or without a language tag. A fence
/// around the whole message is stripped; otherwise the first fenced block in
/// the message is taken, so prose before or after it is dropped.
fn unfence(text: &str) -> &str {
let Some(inner) = text.strip_prefix("```").and_then(|t| t.strip_suffix("```")) else {
return text;
let inner = match text.strip_prefix("```").and_then(|t| t.strip_suffix("```")) {
Some(inner) => inner,
None => match text
.split_once("```")
.and_then(|(_, t)| t.split_once("```"))
{
Some((block, _)) => block,
None => return text,
},
};
let inner = inner.strip_prefix("json").unwrap_or(inner);
inner.trim()
@ -113,6 +122,20 @@ mod tests {
);
}
#[test]
fn a_fenced_answer_inside_prose_parses() {
let a = parse(
"Here is my answer:\n```json\n{\"verdict\":\"match\",\"certainty\":80,\"reason\":\"x\"}\n```\nHope that helps.",
)
.expect("fence in prose");
assert_eq!((a.verdict, a.certainty), (Verdict::Match, 80));
let untagged =
"Answer: ```{\"verdict\":\"no-match\",\"certainty\":70,\"reason\":\"\"}``` done";
assert_eq!(parse(untagged).expect("untagged").verdict, Verdict::NoMatch);
assert!(parse("Here is my answer:\n```json\nnot json\n```").is_err());
assert!(parse("an unclosed ```json\n{\"verdict\":\"match\"").is_err());
}
#[test]
fn a_malformed_answer_is_an_error() {
for bad in [