review-classifier: ask a check once more when its answer isn't valid JSON
An answer that fails to validate is asked for a second time, for that classification only, with a reminder to reply with the JSON object alone. The second ask comes out of the classification's CLASSIFICATION_DEADLINE: it gets what the first ask left, and is skipped when less than 20 seconds remain, so RUN_DEADLINE's bound still holds. A failed ask (runtime error, timeout) is not repeated. If the second answer does not validate either, the row has no answer, as before. Model::ask takes the deadline for the ask; the ACP runtime no longer holds its own. The parser also takes the first fenced block of a reply that has prose around the fence; prose around a bare object is still rejected.
This commit is contained in:
parent
b8897e7c7a
commit
ce7b3e055f
8 changed files with 345 additions and 66 deletions
|
|
@ -54,6 +54,10 @@ pub fn shared_context(run: &TryOut) -> String {
|
|||
out
|
||||
}
|
||||
|
||||
/// Appended to the question when it is asked a second time.
|
||||
const JSON_REMINDER: &str = "Your previous answer was not a JSON object. Reply with the JSON \
|
||||
object and nothing else: no text before or after it, and no code fence.";
|
||||
|
||||
/// One classification's prompt: the run's shared context unchanged, then the
|
||||
/// question.
|
||||
pub struct Prompt<'a> {
|
||||
|
|
@ -67,6 +71,16 @@ impl Prompt<'_> {
|
|||
pub fn text(&self) -> String {
|
||||
format!("{}{}", self.shared, self.question)
|
||||
}
|
||||
|
||||
/// The same prompt with a reminder to answer with the JSON object alone,
|
||||
/// for asking again after an answer that was not that object.
|
||||
#[must_use]
|
||||
pub fn with_json_reminder(&self) -> Self {
|
||||
Self {
|
||||
shared: self.shared,
|
||||
question: format!("{}\n{JSON_REMINDER}\n", self.question.trim_end()),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
|
|
|
|||
Loading…
Reference in a new issue