review classifier: the most classifications a run carries is a setting
The limit was a constant of ten in swarm-wire, so a repo with eleven rules could not be reviewed and nothing short of a rebuild changed it. It is now `max_classifications` in the classifier settings, beside `max_concurrent_reviews`: 1 to 1000, ten until set, and a document stored before the field decodes with the default. The classifier settings form in the swarm UI gets a "rules per review" field, and the try-out's add button stops at the saved limit instead of a hard-coded ten. The controller checks a try-out, and a requested review's rules file, against the setting where the run starts. The rules file itself may hold up to the 1000 the setting allows, so raising the limit needs no change to a repo. A review whose rules exceed the limit posts a no-verdict comment saying so. A run's reply deadline was a constant that covered ten classifications, a minute plus three each. It is now computed from the run's own size, so a bigger run waits as long as it needs and a default-sized one still waits 31 minutes. The classifier's answer window in the queue permissions covers the biggest run the setting allows.
This commit is contained in:
iris
mara
parent
e436d356f3
commit
cffe8895df
15 changed files with 400 additions and 101 deletions
|
|
@ -261,8 +261,9 @@ repo's default branch, so you can run the repo's own rules or start from them
|
|||
and edit. It reads the file each time, so a rule merged a minute ago shows up.
|
||||
The page names the file and the commit it read, and asks before it replaces
|
||||
classifications you have written. If the repo has no rules file, the file
|
||||
doesn't parse, or it holds more than ten rules, the page shows the reason and
|
||||
leaves your classifications as they were.
|
||||
doesn't parse, or it holds more than a thousand rules, the page shows the reason and
|
||||
leaves your classifications as they were. **add a classification** stops at the
|
||||
**rules per review** limit of the classifier settings below, ten until set.
|
||||
|
||||
A run takes a minute or more, and up to half an hour for ten classifications.
|
||||
The page says it's running and shows how long it has been, then lists each classification with its verdict, the model's certainty, the
|
||||
|
|
@ -302,10 +303,14 @@ cache write prices. The provider's fields sit under their own **provider**
|
|||
heading, the URL and the model each on a row of their own and the prices in a
|
||||
grid on the same columns as the fields above. The effort is one of low, medium, high, xhigh and max, or
|
||||
the runtime's own default. Concurrent reviews is a whole number from 1 to 8;
|
||||
try-outs don't count toward it.
|
||||
try-outs don't count toward it. Rules per review is the most classifications
|
||||
one run carries, a whole number from 1 to 1000, ten until you set it. The
|
||||
classifications of a run go one after another, each for up to three minutes, so
|
||||
a high limit means a long run. A repo whose rules file holds more than the
|
||||
limit gets a no-verdict review, and a try-out with more gets a 400.
|
||||
|
||||
**save** is on once something changed and the fields are valid. A save changes
|
||||
the runtime, model, effort and limit, and the provider too while the runtime is
|
||||
the runtime, model, effort and limits, and the provider too while the runtime is
|
||||
ACP, so switching back to API key keeps the stored provider for next time. If
|
||||
the controller refuses it, the page shows the reason and leaves the form as you
|
||||
typed it. A change takes effect from the next run, with no restart. The page
|
||||
|
|
|
|||
|
|
@ -603,7 +603,8 @@ adds to `verdicts` but runs no checks, so it adds nothing to `checks` or
|
|||
as `queue_failed` and a `no_verdict`. `no_answer` is a classification that
|
||||
failed, so a `no_verdict` run with a `no_answer` check is the usual cause of
|
||||
a comment instead of a review. The `check` values are the names in a repo's
|
||||
rules file, at most 10 per repo.
|
||||
rules file, at most the settings' `max_classifications` per repo (ten until
|
||||
set).
|
||||
|
||||
The `review-classifier` dashboard in Grafana charts these.
|
||||
|
||||
|
|
|
|||
|
|
@ -30,7 +30,7 @@ with exactly these four keys:
|
|||
`negative` means a match is a violation
|
||||
|
||||
Any other key, at the top or in an entry, is an error: a misspelt key fails
|
||||
the load instead of vanishing. An empty list, more than ten entries, an empty
|
||||
the load instead of vanishing. An empty list, more than a thousand entries, an empty
|
||||
name or description, and two entries with one name are errors too. A file
|
||||
that exists but fails these checks never falls back to a pass: the run posts
|
||||
no verdict.
|
||||
|
|
@ -215,6 +215,7 @@ replaces it.
|
|||
"model": "haiku",
|
||||
"effort": null,
|
||||
"max_concurrent_reviews": 2,
|
||||
"max_classifications": 10,
|
||||
"runtime": "api-key",
|
||||
"acp": null
|
||||
}
|
||||
|
|
@ -228,6 +229,15 @@ replaces it.
|
|||
offers it
|
||||
- `max_concurrent_reviews`: 1 to 8, how many requested reviews run at once.
|
||||
Try-outs don't count toward it. The rest wait in the queue for a place
|
||||
- `max_classifications`: 1 to 1000, ten by default (also for a document
|
||||
stored before the field existed): the most classifications one run
|
||||
carries. The controller refuses a try-out with more, and a review of a repo
|
||||
whose rules file holds more posts a no-verdict comment saying so. A rules
|
||||
file may hold up to 1000 either way, so raising the limit needs no change to
|
||||
the repo. The classifications of a run go one after another, each for up to
|
||||
three minutes, so the controller waits a minute plus three per
|
||||
classification for the reply (31 minutes for ten), and the classifier may
|
||||
answer for as long as the biggest run the limit allows
|
||||
- `runtime`: `api-key` or `acp` (see [Runtimes](#runtimes))
|
||||
- `acp`: the provider the ACP runtime runs on, or `null`. It stays stored
|
||||
while the runtime is `api-key`, so switching back and forth needs no
|
||||
|
|
@ -255,11 +265,12 @@ does. Every error is problem+json with the reason in `detail`:
|
|||
|
||||
- **400** for a body that isn't JSON or isn't this shape: an unknown field, a
|
||||
`null` other than for `effort` or `acp`, an effort outside the five levels,
|
||||
a `max_concurrent_reviews` that isn't a whole number, a runtime other than
|
||||
a `max_concurrent_reviews` or `max_classifications` that isn't a whole
|
||||
number, a runtime other than
|
||||
the two, or an `acp` without `base_url`, `model` or `prices`. Also **400**
|
||||
when the document the PUT would store breaks a rule: an empty `model`, one
|
||||
with whitespace or one starting with `-`, a `max_concurrent_reviews` outside
|
||||
1 to 8, an `acp.base_url` that isn't `https://`, an empty `acp.model` or one
|
||||
1 to 8, a `max_classifications` outside 1 to 1000, an `acp.base_url` that isn't `https://`, an empty `acp.model` or one
|
||||
with whitespace, a negative price, or the `acp` runtime with no `acp`
|
||||
- **401** for a PUT with no `Remote-User`
|
||||
- **503** when this host has no swarm queue, or it isn't connected
|
||||
|
|
@ -588,7 +599,7 @@ The route answers **201** with `branch`, `path` (the file written) and
|
|||
file changed on the default branch during the push
|
||||
- **422** for a default branch holding both rules files or one the `rules`
|
||||
route refuses; classifications the file already holds unchanged; or a
|
||||
merge that breaks a rule, such as more than ten classifications
|
||||
merge that breaks a rule, such as more than a thousand classifications
|
||||
- **503** when this host has no forge token, the service account's token
|
||||
isn't minted yet, or the forge failed or didn't answer within two minutes
|
||||
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
// <ClassifierSettings> — the review classifier's runtime, model, effort and
|
||||
// how many requested reviews run at once, shown and saved here
|
||||
// how many requested reviews run at once and how many rules a run takes, shown
|
||||
// and saved here
|
||||
// (`GET`/`PUT /api/review-classifier/settings`). A PUT changes only the
|
||||
// fields it carries: the form sends the runtime, model, effort and limit, and
|
||||
// the ACP provider only while the runtime is ACP, so the stored provider
|
||||
|
|
@ -19,7 +20,9 @@ import { TextField } from "../../ui/text-field/TextField.js";
|
|||
import {
|
||||
EFFORTS,
|
||||
MAX_REVIEWS,
|
||||
MAX_RULES,
|
||||
MIN_REVIEWS,
|
||||
MIN_RULES,
|
||||
changed,
|
||||
draftOf,
|
||||
settingsProblem,
|
||||
|
|
@ -177,6 +180,13 @@ export function ClassifierSettings() {
|
|||
pattern="[0-9]+"
|
||||
onInput={(max) => edit({ ...draft, max })}
|
||||
/>
|
||||
<TextField
|
||||
id="classifier-max-rules"
|
||||
label={`rules per review (${MIN_RULES} to ${MAX_RULES})`}
|
||||
value={draft.maxRules}
|
||||
pattern="[0-9]+"
|
||||
onInput={(maxRules) => edit({ ...draft, maxRules })}
|
||||
/>
|
||||
</FieldGrid>
|
||||
{draft.runtime === "acp" ? (
|
||||
<fieldset class="ui-classifier-settings-group">
|
||||
|
|
|
|||
|
|
@ -33,6 +33,8 @@ export interface ClassifierSettingsBody {
|
|||
* `--effort`, or opencode's. */
|
||||
effort: Effort | null;
|
||||
max_concurrent_reviews: number;
|
||||
/** The most classifications (rules) one run carries. */
|
||||
max_classifications: number;
|
||||
runtime: Runtime;
|
||||
/** The provider record. It stays stored while the runtime is `api-key`. */
|
||||
acp: AcpSettings | null;
|
||||
|
|
@ -44,6 +46,8 @@ export interface SettingsDraft {
|
|||
/** An effort, or `""` for the runtime's own default. */
|
||||
effort: string;
|
||||
max: string;
|
||||
/** The most classifications one run carries. */
|
||||
maxRules: string;
|
||||
runtime: Runtime;
|
||||
acpUrl: string;
|
||||
acpModel: string;
|
||||
|
|
@ -56,6 +60,8 @@ export interface SettingsDraft {
|
|||
|
||||
export const MIN_REVIEWS = 1;
|
||||
export const MAX_REVIEWS = 8;
|
||||
export const MIN_RULES = 1;
|
||||
export const MAX_RULES = 1000;
|
||||
|
||||
export function draftOf(s: ClassifierSettingsBody): SettingsDraft {
|
||||
const acp = s.acp;
|
||||
|
|
@ -63,6 +69,7 @@ export function draftOf(s: ClassifierSettingsBody): SettingsDraft {
|
|||
model: s.model,
|
||||
effort: s.effort ?? "",
|
||||
max: String(s.max_concurrent_reviews),
|
||||
maxRules: String(s.max_classifications),
|
||||
runtime: s.runtime,
|
||||
acpUrl: acp?.base_url ?? "",
|
||||
acpModel: acp?.model ?? "",
|
||||
|
|
@ -86,6 +93,15 @@ export function maxValue(text: string): number | null {
|
|||
return n >= MIN_REVIEWS && n <= MAX_REVIEWS ? n : null;
|
||||
}
|
||||
|
||||
/** The rules limit as a number, or null when it is not a whole number in
|
||||
* range. */
|
||||
export function rulesValue(text: string): number | null {
|
||||
const t = text.trim();
|
||||
if (!/^\d+$/.test(t)) return null;
|
||||
const n = Number(t);
|
||||
return n >= MIN_RULES && n <= MAX_RULES ? n : null;
|
||||
}
|
||||
|
||||
/** A price as a number, or null when it is not a number of 0 or more. */
|
||||
export function priceValue(text: string): number | null {
|
||||
const t = text.trim();
|
||||
|
|
@ -124,6 +140,9 @@ export function settingsProblem(d: SettingsDraft): string | null {
|
|||
if (maxValue(d.max) === null) {
|
||||
return `concurrent reviews is a whole number from ${MIN_REVIEWS} to ${MAX_REVIEWS}`;
|
||||
}
|
||||
if (rulesValue(d.maxRules) === null) {
|
||||
return `rules per review is a whole number from ${MIN_RULES} to ${MAX_RULES}`;
|
||||
}
|
||||
return d.runtime === "acp" ? acpProblem(d) : null;
|
||||
}
|
||||
|
||||
|
|
@ -153,6 +172,7 @@ export function toSettings(
|
|||
model: d.model.trim(),
|
||||
effort: d.effort === "" ? null : (d.effort as Effort),
|
||||
max_concurrent_reviews: maxValue(d.max) ?? MIN_REVIEWS,
|
||||
max_classifications: rulesValue(d.maxRules) ?? MIN_RULES,
|
||||
runtime: d.runtime,
|
||||
};
|
||||
return d.runtime === "acp" ? { ...body, acp: toAcp(d) } : body;
|
||||
|
|
@ -169,6 +189,7 @@ export function changed(
|
|||
d.model.trim() !== now.model ||
|
||||
d.effort !== now.effort ||
|
||||
d.max.trim() !== now.max ||
|
||||
d.maxRules.trim() !== now.maxRules ||
|
||||
d.runtime !== now.runtime;
|
||||
if (base) return true;
|
||||
// The provider fields only save on the ACP runtime.
|
||||
|
|
@ -185,12 +206,16 @@ export function changed(
|
|||
|
||||
/** The settings in force as one short line, for a folded view of them: the
|
||||
* runtime and the model it runs, the effort when one is set, and how many
|
||||
* reviews run at once. The model is the provider's under ACP. */
|
||||
* reviews run at once and how many rules a run takes. The model is the
|
||||
* provider's under ACP. */
|
||||
export function settingsSummary(s: ClassifierSettingsBody): string {
|
||||
const model = s.runtime === "acp" ? (s.acp?.model ?? "") : s.model;
|
||||
const parts = [s.runtime, model, s.effort ? `${s.effort} effort` : ""].filter(
|
||||
(p) => p !== "",
|
||||
);
|
||||
parts.push(`${s.max_concurrent_reviews} at once`);
|
||||
parts.push(
|
||||
`${s.max_concurrent_reviews} at once`,
|
||||
`up to ${s.max_classifications} rules`,
|
||||
);
|
||||
return parts.join(" · ");
|
||||
}
|
||||
|
|
|
|||
|
|
@ -55,6 +55,7 @@ import "./ReviewTryOut.css";
|
|||
const TRY_OUT = "/api/review-classifier/try-out";
|
||||
const RULES = "/api/review-classifier/rules";
|
||||
const OPEN_PR = "/api/review-classifier/open-pr";
|
||||
const SETTINGS = "/api/review-classifier/settings";
|
||||
const PULL_REPOS = "/api/pulls/repos";
|
||||
const PULLS = "/api/pulls";
|
||||
// How long the repo field rests before its PRs are fetched, so typing a repo
|
||||
|
|
@ -111,6 +112,9 @@ export function ReviewTryOut() {
|
|||
useState<ProblemDetails | null>(null);
|
||||
const [pulls, setPulls] = useState<OpenPull[]>([]);
|
||||
const [pullsError, setPullsError] = useState<ProblemDetails | null>(null);
|
||||
const [maxClassifications, setMaxClassifications] = useState<number | null>(
|
||||
null,
|
||||
);
|
||||
|
||||
const running = starting || runId !== null;
|
||||
|
||||
|
|
@ -139,6 +143,27 @@ export function ReviewTryOut() {
|
|||
};
|
||||
}, []);
|
||||
|
||||
// How many rows a run takes: the classifier settings', read once. Until it
|
||||
// is known, or when it cannot be read, the run endpoint is the only cap.
|
||||
useEffect(() => {
|
||||
let cancelled = false;
|
||||
void (async () => {
|
||||
try {
|
||||
const res = await fetch(SETTINGS);
|
||||
if (cancelled || !res.ok) return;
|
||||
const body = (await res.json()) as { max_classifications?: number };
|
||||
if (!cancelled && typeof body.max_classifications === "number") {
|
||||
setMaxClassifications(body.max_classifications);
|
||||
}
|
||||
} catch {
|
||||
// The endpoint still refuses a run that is too long.
|
||||
}
|
||||
})();
|
||||
return () => {
|
||||
cancelled = true;
|
||||
};
|
||||
}, []);
|
||||
|
||||
// The open PRs of the repo once it is typed or picked and has stopped
|
||||
// changing. A repo that is not valid yet has no list.
|
||||
useEffect(() => {
|
||||
|
|
@ -224,7 +249,7 @@ export function ReviewTryOut() {
|
|||
|
||||
const openPrProblem = rowsProblem(repo, drafts);
|
||||
const problem = runProblem(repo, pr, drafts);
|
||||
const capProblem = addProblem(drafts.length);
|
||||
const capProblem = addProblem(drafts.length, maxClassifications);
|
||||
|
||||
async function run(e: Event) {
|
||||
e.preventDefault();
|
||||
|
|
|
|||
|
|
@ -150,14 +150,13 @@ export function duplicateName(drafts: ClassificationDraft[]): string | null {
|
|||
return null;
|
||||
}
|
||||
|
||||
/** More classifications than this are not offered a row. It is the try-out
|
||||
* endpoint's own cap, which answers a longer list with a 400. */
|
||||
export const MAX_CLASSIFICATIONS = 10;
|
||||
|
||||
/** Why another row cannot be added, or null while there is room. */
|
||||
export function addProblem(count: number): string | null {
|
||||
return count >= MAX_CLASSIFICATIONS
|
||||
? `a run takes at most ${MAX_CLASSIFICATIONS} classifications`
|
||||
/** Why another row cannot be added, or null while there is room. `max` is the
|
||||
* classifier settings' `max_classifications`, the cap the try-out endpoint
|
||||
* answers a longer list with a 400 for; null while the settings have not
|
||||
* loaded, when the endpoint stays the only gate. */
|
||||
export function addProblem(count: number, max: number | null): string | null {
|
||||
return max !== null && count >= max
|
||||
? `a run takes at most ${max} classifications, as the classifier's settings say`
|
||||
: null;
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -8,6 +8,7 @@ import {
|
|||
changed,
|
||||
draftOf,
|
||||
maxValue,
|
||||
rulesValue,
|
||||
priceValue,
|
||||
settingsProblem,
|
||||
settingsSummary,
|
||||
|
|
@ -18,6 +19,7 @@ const apiKey = {
|
|||
model: "haiku",
|
||||
effort: null,
|
||||
max_concurrent_reviews: 2,
|
||||
max_classifications: 10,
|
||||
runtime: "api-key",
|
||||
acp: null,
|
||||
};
|
||||
|
|
@ -57,6 +59,7 @@ test("the API key request leaves the provider out, so the stored one stays", ()
|
|||
model: "haiku",
|
||||
effort: null,
|
||||
max_concurrent_reviews: 2,
|
||||
max_classifications: 10,
|
||||
runtime: "api-key",
|
||||
});
|
||||
assert.equal("acp" in body, false);
|
||||
|
|
@ -119,14 +122,33 @@ test("the reviews limit and a price parse as the form needs", () => {
|
|||
assert.equal(priceValue("1e3"), null);
|
||||
});
|
||||
|
||||
test("the summary names the runtime, model and how many run at once", () => {
|
||||
assert.equal(settingsSummary(apiKey), "api-key · haiku · 2 at once");
|
||||
test("the summary names the runtime, model, how many run at once and the rules a run takes", () => {
|
||||
assert.equal(
|
||||
settingsSummary(apiKey),
|
||||
"api-key · haiku · 2 at once · up to 10 rules",
|
||||
);
|
||||
});
|
||||
|
||||
test("the rules limit parses and checks as the form needs", () => {
|
||||
assert.equal(rulesValue("1"), 1);
|
||||
assert.equal(rulesValue(" 1000 "), 1000);
|
||||
assert.equal(rulesValue("0"), null);
|
||||
assert.equal(rulesValue("1001"), null);
|
||||
assert.equal(rulesValue("2.5"), null);
|
||||
assert.equal(rulesValue(""), null);
|
||||
const d = { ...draftOf(apiKey), maxRules: "5000" };
|
||||
assert.match(settingsProblem(d), /rules per review.*1 to 1000/);
|
||||
assert.equal(
|
||||
toSettings({ ...draftOf(apiKey), maxRules: "40" }).max_classifications,
|
||||
40,
|
||||
);
|
||||
assert.equal(changed({ ...draftOf(apiKey), maxRules: "11" }, apiKey), true);
|
||||
});
|
||||
|
||||
test("the summary names an effort when one is set", () => {
|
||||
assert.equal(
|
||||
settingsSummary({ ...apiKey, effort: "high" }),
|
||||
"api-key · haiku · high effort · 2 at once",
|
||||
"api-key · haiku · high effort · 2 at once · up to 10 rules",
|
||||
);
|
||||
});
|
||||
|
||||
|
|
@ -141,6 +163,6 @@ test("under ACP the summary names the provider's model, not the claude one", ()
|
|||
prices: { input: 1, output: 2 },
|
||||
},
|
||||
}),
|
||||
"acp · glm · 2 at once",
|
||||
"acp · glm · 2 at once · up to 10 rules",
|
||||
);
|
||||
});
|
||||
|
|
|
|||
|
|
@ -5,7 +5,6 @@
|
|||
import assert from "node:assert/strict";
|
||||
import { test } from "node:test";
|
||||
import {
|
||||
MAX_CLASSIFICATIONS,
|
||||
addProblem,
|
||||
draftFromRule,
|
||||
draftProblem,
|
||||
|
|
@ -19,10 +18,16 @@ import {
|
|||
toBody,
|
||||
} from "../src/pages/review-tryout/TryOutTypes.ts";
|
||||
|
||||
test("a row is offered until the endpoint's cap is reached", () => {
|
||||
assert.equal(addProblem(1), null);
|
||||
assert.equal(addProblem(MAX_CLASSIFICATIONS - 1), null);
|
||||
assert.match(addProblem(MAX_CLASSIFICATIONS), /at most 10 classifications/);
|
||||
test("a row is offered until the settings' cap is reached", () => {
|
||||
assert.equal(addProblem(1, 10), null);
|
||||
assert.equal(addProblem(9, 10), null);
|
||||
assert.match(addProblem(10, 10), /at most 10 classifications/);
|
||||
assert.match(addProblem(25, 25), /at most 25 classifications/);
|
||||
});
|
||||
|
||||
test("with the cap unknown, every row is offered and the endpoint stays the gate", () => {
|
||||
assert.equal(addProblem(10, null), null);
|
||||
assert.equal(addProblem(500, null), null);
|
||||
});
|
||||
|
||||
// A row ready to send, with `over` replacing any of its fields.
|
||||
|
|
|
|||
|
|
@ -32,7 +32,7 @@ use axum::http::{HeaderMap, StatusCode};
|
|||
use problem_details::ProblemDetails;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use swarm_wire::review_classifier::{
|
||||
Classification, MAX_CLASSIFICATIONS, Overall, RUN_DEADLINE, Reply, Row, SUBJECT, TryOut,
|
||||
CLASSIFICATION_LIMITS, Classification, Overall, Reply, Row, SUBJECT, TryOut, run_deadline,
|
||||
};
|
||||
use utoipa::{IntoParams, ToSchema};
|
||||
|
||||
|
|
@ -114,9 +114,8 @@ struct Rule {
|
|||
unknown: BTreeMap<String, serde_json::Value>,
|
||||
}
|
||||
|
||||
/// `running` lasts at most the run deadline, 31 minutes: ten
|
||||
/// classifications at three minutes each, one after another, and a minute
|
||||
/// more.
|
||||
/// `running` lasts at most the run deadline: the classifications at three
|
||||
/// minutes each, one after another, and a minute more (31 minutes for ten).
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, ToSchema)]
|
||||
#[serde(rename_all = "lowercase")]
|
||||
pub(crate) enum RunState {
|
||||
|
|
@ -177,6 +176,8 @@ pub(crate) struct TryOuts {
|
|||
struct Entry {
|
||||
status: TryOutStatus,
|
||||
started: Instant,
|
||||
/// The longest the run's reply can take: [`run_deadline`] of its size.
|
||||
deadline: Duration,
|
||||
settled: Option<Instant>,
|
||||
}
|
||||
|
||||
|
|
@ -184,7 +185,7 @@ impl Entry {
|
|||
/// Expired once [`RESULT_TTL`] has passed since the reply, or since the
|
||||
/// latest a reply can come.
|
||||
fn expired(&self, now: Instant) -> bool {
|
||||
let since = self.settled.unwrap_or(self.started + RUN_DEADLINE);
|
||||
let since = self.settled.unwrap_or(self.started + self.deadline);
|
||||
now.saturating_duration_since(since) > RESULT_TTL
|
||||
}
|
||||
}
|
||||
|
|
@ -196,7 +197,7 @@ impl TryOuts {
|
|||
.unwrap_or_else(std::sync::PoisonError::into_inner)
|
||||
}
|
||||
|
||||
fn insert(&self, id: String, head_sha: String) {
|
||||
fn insert(&self, id: String, head_sha: String, deadline: Duration) {
|
||||
let now = Instant::now();
|
||||
let mut runs = self.lock();
|
||||
runs.retain(|_, e| !e.expired(now));
|
||||
|
|
@ -211,6 +212,7 @@ impl TryOuts {
|
|||
error: None,
|
||||
},
|
||||
started: now,
|
||||
deadline,
|
||||
settled: None,
|
||||
},
|
||||
);
|
||||
|
|
@ -270,18 +272,20 @@ impl FileSource for forge::Client {
|
|||
}
|
||||
}
|
||||
|
||||
/// Where a run is sent: the queue in production, a fake in the tests.
|
||||
/// Where a run is sent: the queue in production, a fake in the tests. The
|
||||
/// reply is waited for at most `deadline`, the [`run_deadline`] of the run's
|
||||
/// size.
|
||||
pub(crate) trait RunSink: Clone + Send + Sync + 'static {
|
||||
fn run(&self, request: Vec<u8>) -> impl Future<Output = Reply> + Send;
|
||||
fn run(&self, request: Vec<u8>, deadline: Duration) -> impl Future<Output = Reply> + Send;
|
||||
}
|
||||
|
||||
impl RunSink for async_nats::Client {
|
||||
async fn run(&self, request: Vec<u8>) -> Reply {
|
||||
async fn run(&self, request: Vec<u8>, deadline: Duration) -> Reply {
|
||||
use async_nats::RequestErrorKind;
|
||||
|
||||
let sent = async_nats::Request::new()
|
||||
.payload(request.into())
|
||||
.timeout(Some(RUN_DEADLINE));
|
||||
.timeout(Some(deadline));
|
||||
let error = match self.send_request(SUBJECT, sent).await {
|
||||
Ok(msg) => {
|
||||
return serde_json::from_slice(&msg.payload).unwrap_or_else(|e| Reply::Failed {
|
||||
|
|
@ -296,7 +300,7 @@ impl RunSink for async_nats::Client {
|
|||
}
|
||||
RequestErrorKind::TimedOut => format!(
|
||||
"the review classifier sent no answer within {} minutes",
|
||||
RUN_DEADLINE.as_secs() / 60
|
||||
deadline.as_secs() / 60
|
||||
),
|
||||
_ => format!("sending the run to the review classifier failed: {e}"),
|
||||
},
|
||||
|
|
@ -316,7 +320,8 @@ impl RunSink for async_nats::Client {
|
|||
(status = 400, description = "the body is not a valid run: not JSON or not this \
|
||||
shape (an unknown polarity, a negative or non-integer number, a threshold over \
|
||||
255); a repo that is not owner/name with segments of `[A-Za-z0-9._-]` other than \
|
||||
`.` and `..`; a PR number of 0; no classification or more than ten; a \
|
||||
`.` and `..`; a PR number of 0; no classification or more than the settings' \
|
||||
`max_classifications` (ten until set); a \
|
||||
classification whose trimmed name or description is empty; two with one trimmed \
|
||||
name; or a threshold over 100 (problem+json)", body = String),
|
||||
(status = 401, description = "the request carries no `Remote-User`, so there is no \
|
||||
|
|
@ -398,8 +403,9 @@ pub(crate) async fn post_try_out(
|
|||
path = "/api/review-classifier/try-out/{id}",
|
||||
params(("id" = String, Path, description = "the id the POST answered with")),
|
||||
responses(
|
||||
(status = 200, description = "the run: `running` for at most 31 minutes, then \
|
||||
`done` or `failed`", body = TryOutStatus),
|
||||
(status = 200, description = "the run: `running` for at most a minute plus three \
|
||||
per classification (31 minutes for ten), then `done` or `failed`",
|
||||
body = TryOutStatus),
|
||||
(status = 404, description = "no run has this id: it never existed, its result \
|
||||
expired an hour after it settled, or the controller restarted since the POST, \
|
||||
which loses every run (problem+json)", body = String),
|
||||
|
|
@ -436,7 +442,7 @@ pub(crate) async fn get_try_out(
|
|||
not parse as its extension says; or it holds a key other than `classifications`, \
|
||||
or a classification with a key other than `name`, `description`, `polarity` and \
|
||||
`threshold`; or it fails a check the try-out POST makes: no classification or \
|
||||
more than ten, an empty trimmed name or description, two with one trimmed name, \
|
||||
more than 1000, an empty trimmed name or description, two with one trimmed name, \
|
||||
or a threshold over 100 (problem+json)", body = String),
|
||||
(status = 503, description = "this host has no forge token yet; or the forge failed \
|
||||
or did not answer within two minutes (problem+json)", body = String),
|
||||
|
|
@ -566,7 +572,10 @@ fn parse_rules(path: &str, bytes: &[u8]) -> Result<Vec<Classification>, String>
|
|||
}
|
||||
classifications.push(rule.classification);
|
||||
}
|
||||
check_classifications(&classifications).map_err(|e| format!("{path}: {e}"))?;
|
||||
// The file may hold up to the most the setting allows; the setting itself
|
||||
// is checked where a run starts.
|
||||
check_classifications(&classifications, *CLASSIFICATION_LIMITS.end())
|
||||
.map_err(|e| format!("{path}: {e}"))?;
|
||||
Ok(classifications)
|
||||
}
|
||||
|
||||
|
|
@ -587,7 +596,7 @@ impl Run {
|
|||
.ok()
|
||||
.filter(|n| *n > 0)
|
||||
.ok_or_else(|| format!("{} is not a PR number", req.pr))?;
|
||||
check_classifications(&req.classifications)?;
|
||||
check_classifications(&req.classifications, *CLASSIFICATION_LIMITS.end())?;
|
||||
Ok(Self {
|
||||
owner: owner.to_owned(),
|
||||
name: name.to_owned(),
|
||||
|
|
@ -604,14 +613,22 @@ pub(crate) fn split_repo(repo: &str) -> Result<(&str, &str), String> {
|
|||
.ok_or_else(|| format!("repo {repo:?} is not owner/name"))
|
||||
}
|
||||
|
||||
/// What is wrong with `classifications` as one run's, if anything.
|
||||
fn check_classifications(classifications: &[Classification]) -> Result<(), String> {
|
||||
let count = classifications.len();
|
||||
if count == 0 || count > MAX_CLASSIFICATIONS {
|
||||
/// Whether a run of `count` classifications is within the `max` the settings
|
||||
/// allow, or what is wrong with it.
|
||||
pub(crate) fn check_count(count: usize, max: usize) -> Result<(), String> {
|
||||
if count == 0 || count > max {
|
||||
return Err(format!(
|
||||
"{count} classifications: a run carries 1 to {MAX_CLASSIFICATIONS}"
|
||||
"{count} classifications: a run carries 1 to {max}, as the review classifier's \
|
||||
settings allow"
|
||||
));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// What is wrong with `classifications` as one run's of at most `max`, if
|
||||
/// anything.
|
||||
fn check_classifications(classifications: &[Classification], max: usize) -> Result<(), String> {
|
||||
check_count(classifications.len(), max)?;
|
||||
let mut names = HashSet::new();
|
||||
for c in classifications {
|
||||
let name = c.name.trim();
|
||||
|
|
@ -665,6 +682,9 @@ async fn start(
|
|||
settings: Settings,
|
||||
) -> Result<(StatusCode, Json<TryOutAccepted>), ProblemDetails> {
|
||||
let started_unix = crate::agent_renewal::unix_now();
|
||||
check_count(run.classifications.len(), settings.max_classifications)
|
||||
.map_err(|d| error_problem(StatusCode::BAD_REQUEST, &d))?;
|
||||
let deadline = run_deadline(run.classifications.len());
|
||||
let runtime = settings
|
||||
.runtime()
|
||||
.map_err(|e| error_problem(StatusCode::INTERNAL_SERVER_ERROR, &e))?;
|
||||
|
|
@ -698,7 +718,7 @@ async fn start(
|
|||
)
|
||||
})?;
|
||||
let id = mint_id().map_err(|e| error_problem(StatusCode::INTERNAL_SERVER_ERROR, &e))?;
|
||||
try_outs.insert(id.clone(), pull.head_sha.clone());
|
||||
try_outs.insert(id.clone(), pull.head_sha.clone(), deadline);
|
||||
let try_outs = try_outs.clone();
|
||||
let (settle_id, repo, pr, requester) = (
|
||||
id.clone(),
|
||||
|
|
@ -707,7 +727,7 @@ async fn start(
|
|||
request.operator.clone(),
|
||||
);
|
||||
tokio::spawn(async move {
|
||||
let reply = sink.run(payload).await;
|
||||
let reply = sink.run(payload, deadline).await;
|
||||
let (rows, overall) = runs::answer(&reply);
|
||||
let (outcome, detail) = match &reply {
|
||||
Reply::Done { .. } => (runs::RunOutcome::Done, None),
|
||||
|
|
@ -910,7 +930,7 @@ pub(crate) mod tests {
|
|||
}
|
||||
|
||||
impl RunSink for Fake {
|
||||
async fn run(&self, request: Vec<u8>) -> Reply {
|
||||
async fn run(&self, request: Vec<u8>, _deadline: Duration) -> Reply {
|
||||
if let Ok(sent) = serde_json::from_slice(&request) {
|
||||
self.sent.lock().expect("unpoisoned").push(sent);
|
||||
}
|
||||
|
|
@ -1032,6 +1052,7 @@ pub(crate) mod tests {
|
|||
model: "opus".to_owned(),
|
||||
effort: Some(Effort::High),
|
||||
max_concurrent_reviews: 2,
|
||||
max_classifications: 10,
|
||||
runtime: settings::RuntimeKind::Acp,
|
||||
acp: Some(acp.clone()),
|
||||
},
|
||||
|
|
@ -1051,6 +1072,42 @@ pub(crate) mod tests {
|
|||
assert_eq!(sent[0].runtime, Runtime::Acp(acp));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_try_out_takes_as_many_classifications_as_the_settings_allow() {
|
||||
let many = |count: usize| {
|
||||
let mut b: serde_json::Value = serde_json::from_str(&body("o/r", 7)).expect("json");
|
||||
let one = b["classifications"][0].clone();
|
||||
b["classifications"] = (0..count)
|
||||
.map(|i| {
|
||||
let mut c = one.clone();
|
||||
c["name"] = format!("c{i}").into();
|
||||
c
|
||||
})
|
||||
.collect();
|
||||
b.to_string()
|
||||
};
|
||||
let (mut state, _) = crate::routes::tests::state_with_roster();
|
||||
state.try_outs.fake = Some(Arc::new(Fake {
|
||||
settings: Settings {
|
||||
max_classifications: 12,
|
||||
..Settings::default()
|
||||
},
|
||||
..Fake::new(done(), Vec::new())
|
||||
}));
|
||||
let app = crate::startup::build_app(state);
|
||||
// Past the default of ten, up to the setting.
|
||||
let (status, accepted) = post(&app, many(12)).await;
|
||||
assert_eq!(status, StatusCode::ACCEPTED, "{accepted}");
|
||||
let (status, problem) = post(&app, many(13)).await;
|
||||
assert_eq!(status, StatusCode::BAD_REQUEST, "{problem}");
|
||||
assert!(
|
||||
problem["detail"]
|
||||
.as_str()
|
||||
.is_some_and(|d| d.contains("13 classifications") && d.contains("1 to 12")),
|
||||
"{problem}"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_failed_run_reports_its_error_and_no_verdict() {
|
||||
let app = app(Reply::Failed {
|
||||
|
|
@ -1182,7 +1239,7 @@ pub(crate) mod tests {
|
|||
|
||||
let nats = crate::test_nats::TestNats::start();
|
||||
let client = nats.client().await;
|
||||
let unanswered = client.run(b"{}".to_vec()).await;
|
||||
let unanswered = client.run(b"{}".to_vec(), run_deadline(1)).await;
|
||||
assert!(
|
||||
matches!(&unanswered, Reply::Failed { error } if error.contains("deploy.review-classifier.enable")),
|
||||
"{unanswered:?}"
|
||||
|
|
@ -1216,7 +1273,7 @@ pub(crate) mod tests {
|
|||
.await
|
||||
.expect("publish");
|
||||
});
|
||||
assert_eq!(client.run(b"{}".to_vec()).await, done());
|
||||
assert_eq!(client.run(b"{}".to_vec(), run_deadline(1)).await, done());
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
|
@ -1225,13 +1282,16 @@ pub(crate) mod tests {
|
|||
let entry = |settled: Option<Instant>| Entry {
|
||||
status: TryOutStatus::settled("abc".to_owned(), done()),
|
||||
started: now,
|
||||
deadline: run_deadline(10),
|
||||
settled,
|
||||
};
|
||||
let deadline = run_deadline(10);
|
||||
assert!(!entry(Some(now)).expired(now + RESULT_TTL));
|
||||
assert!(entry(Some(now)).expired(now + RESULT_TTL + Duration::from_secs(1)));
|
||||
// A run whose reply never came stays until the latest it could have.
|
||||
assert!(!entry(None).expired(now + RUN_DEADLINE + RESULT_TTL));
|
||||
assert!(entry(None).expired(now + RUN_DEADLINE + RESULT_TTL + Duration::from_secs(1)));
|
||||
// A run whose reply never came stays until the latest it could have,
|
||||
// which is later for a bigger run.
|
||||
assert!(!entry(None).expired(now + deadline + RESULT_TTL));
|
||||
assert!(entry(None).expired(now + deadline + RESULT_TTL + Duration::from_secs(1)));
|
||||
}
|
||||
|
||||
/// A rules file holding `classifications`, and `extra` beside them.
|
||||
|
|
@ -1257,11 +1317,22 @@ pub(crate) mod tests {
|
|||
|
||||
/// A JSON rules file holding the one `datetime` classification.
|
||||
pub(crate) fn rules_json() -> (&'static str, String) {
|
||||
rules_json_of(1)
|
||||
}
|
||||
|
||||
/// A rules file holding `count` classifications.
|
||||
pub(crate) fn rules_json_of(count: usize) -> (&'static str, String) {
|
||||
let [_, json] = RULES_PATHS;
|
||||
(
|
||||
json,
|
||||
rules_value(&serde_json::json!([datetime()]), &[]).to_string(),
|
||||
)
|
||||
let rules: Vec<_> = (0..count)
|
||||
.map(|i| {
|
||||
let mut c = datetime();
|
||||
if i > 0 {
|
||||
c["name"] = format!("c{i}").into();
|
||||
}
|
||||
c
|
||||
})
|
||||
.collect();
|
||||
(json, rules_value(&rules.into(), &[]).to_string())
|
||||
}
|
||||
|
||||
/// `file` as both formats, each with its path.
|
||||
|
|
@ -1356,7 +1427,7 @@ classifications:
|
|||
.as_object_mut()
|
||||
.expect("an object")
|
||||
.remove("polarity");
|
||||
let eleven: Vec<_> = (0..11)
|
||||
let too_many: Vec<_> = (0..1001)
|
||||
.map(|i| {
|
||||
let mut c = datetime();
|
||||
c["name"] = format!("c{i}").into();
|
||||
|
|
@ -1381,9 +1452,9 @@ classifications:
|
|||
"threshhold",
|
||||
),
|
||||
(
|
||||
"eleven classifications",
|
||||
rules_value(&eleven.into(), &[]),
|
||||
"11 classifications",
|
||||
"more classifications than any setting allows",
|
||||
rules_value(&too_many.into(), &[]),
|
||||
"1001 classifications",
|
||||
),
|
||||
(
|
||||
"a threshold over 100",
|
||||
|
|
|
|||
|
|
@ -26,7 +26,7 @@ use axum::http::{HeaderMap, StatusCode};
|
|||
use forgejo_api::{ApiErrorKind, ForgejoError};
|
||||
use problem_details::ProblemDetails;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use swarm_wire::review_classifier::{Classification, Polarity};
|
||||
use swarm_wire::review_classifier::{CLASSIFICATION_LIMITS, Classification, Polarity};
|
||||
use utoipa::ToSchema;
|
||||
|
||||
use super::{FileSource, RULES_PATHS, check_classifications, operator, parse_rules, split_repo};
|
||||
|
|
@ -109,7 +109,7 @@ impl BranchSink for forge::Client {
|
|||
commit was made (problem+json)", body = String),
|
||||
(status = 422, description = "the default branch holds both rules files, or one the \
|
||||
`rules` GET refuses; the rules already hold every classification as it is; or \
|
||||
the merged rules fail the try-out POST's checks, such as more than ten \
|
||||
the merged rules fail the try-out POST's checks, such as more than a thousand \
|
||||
(problem+json)", body = String),
|
||||
(status = 503, description = "this host has no forge token yet, or the classifier's \
|
||||
account has none in the store; or the forge failed or did not answer within two \
|
||||
|
|
@ -132,7 +132,7 @@ pub(crate) async fn post_open_pr(
|
|||
let Json(req) = body.map_err(|e| error_problem(StatusCode::BAD_REQUEST, &e.body_text()))?;
|
||||
let (owner, name) =
|
||||
split_repo(&req.repo).map_err(|d| error_problem(StatusCode::BAD_REQUEST, &d))?;
|
||||
check_classifications(&req.classifications)
|
||||
check_classifications(&req.classifications, *CLASSIFICATION_LIMITS.end())
|
||||
.map_err(|d| error_problem(StatusCode::BAD_REQUEST, &d))?;
|
||||
#[cfg(test)]
|
||||
if let Some(fake) = state.try_outs.fake.clone() {
|
||||
|
|
@ -355,7 +355,8 @@ fn merge(existing: Option<&BranchFile>, rows: Vec<Classification>) -> Result<Mer
|
|||
"{path} already holds every classification as it is"
|
||||
));
|
||||
}
|
||||
check_classifications(&merged).map_err(|e| format!("{path} with these merged: {e}"))?;
|
||||
check_classifications(&merged, *CLASSIFICATION_LIMITS.end())
|
||||
.map_err(|e| format!("{path} with these merged: {e}"))?;
|
||||
let content = if std::path::Path::new(&path)
|
||||
.extension()
|
||||
.is_some_and(|e| e == "json")
|
||||
|
|
@ -659,11 +660,11 @@ classifications:
|
|||
let before = parse_rules(yml, OWN.as_bytes()).expect("this repo's rules parse");
|
||||
let err = merge(Some(&file(yml, OWN)), before.clone()).expect_err("nothing to change");
|
||||
assert!(err.contains("already holds"), "{err}");
|
||||
let many: Vec<_> = (0..=10 - before.len())
|
||||
let many: Vec<_> = (0..=1000 - before.len())
|
||||
.map(|i| rule(&format!("extra-{i}"), "d"))
|
||||
.collect();
|
||||
let err = merge(Some(&file(yml, OWN)), many).expect_err("eleven");
|
||||
assert!(err.contains("11 classifications"), "{err}");
|
||||
let err = merge(Some(&file(yml, OWN)), many).expect_err("1001");
|
||||
assert!(err.contains("1001 classifications"), "{err}");
|
||||
let err = merge(Some(&file(yml, "nope: 1")), vec![rule("a", "b")]).expect_err("bad");
|
||||
assert!(err.contains(yml), "{err}");
|
||||
}
|
||||
|
|
|
|||
|
|
@ -36,6 +36,7 @@ use futures_util::StreamExt as _;
|
|||
use serde::{Deserialize, Serialize};
|
||||
use swarm_wire::review_classifier::{
|
||||
Overall, REQUEST_CONSUMER, REQUEST_STREAM, REQUEST_SUBJECT_PREFIX, Reply, Row, TryOut, Verdict,
|
||||
run_deadline,
|
||||
};
|
||||
use tokio::sync::Notify;
|
||||
use tokio::task::JoinSet;
|
||||
|
|
@ -821,7 +822,7 @@ impl ReviewPoster for forge::Client {
|
|||
struct Queue(Option<async_nats::Client>);
|
||||
|
||||
impl RunSink for Queue {
|
||||
async fn run(&self, request: Vec<u8>) -> Reply {
|
||||
async fn run(&self, request: Vec<u8>, deadline: Duration) -> Reply {
|
||||
let Some(client) = &self.0 else {
|
||||
return Reply::Failed {
|
||||
error: "no swarm queue is configured on this host".to_owned(),
|
||||
|
|
@ -832,7 +833,7 @@ impl RunSink for Queue {
|
|||
error: swarm_queue_client::chain(&e),
|
||||
};
|
||||
}
|
||||
client.run(request).await
|
||||
client.run(request, deadline).await
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -1087,6 +1088,14 @@ async fn classify(
|
|||
);
|
||||
}
|
||||
};
|
||||
if let Err(why) = super::check_count(rules.classifications.len(), settings.max_classifications)
|
||||
{
|
||||
return (
|
||||
head,
|
||||
narrowed,
|
||||
Outcome::NotRun(format!("{repo}'s rules do not fit a run: {why}")),
|
||||
);
|
||||
}
|
||||
let runtime = match settings.runtime() {
|
||||
Ok(runtime) => runtime,
|
||||
Err(e) => {
|
||||
|
|
@ -1113,7 +1122,7 @@ async fn classify(
|
|||
let outcome = match serde_json::to_vec(&request) {
|
||||
Ok(payload) => Outcome::Ran {
|
||||
sent,
|
||||
reply: sink.run(payload).await,
|
||||
reply: sink.run(payload, run_deadline(sent)).await,
|
||||
},
|
||||
Err(e) => Outcome::NotRun(format!("encoding the run failed: {e}")),
|
||||
};
|
||||
|
|
@ -1602,7 +1611,7 @@ mod tests {
|
|||
}
|
||||
|
||||
impl RunSink for Gate {
|
||||
async fn run(&self, _request: Vec<u8>) -> Reply {
|
||||
async fn run(&self, _request: Vec<u8>, _deadline: Duration) -> Reply {
|
||||
self.reached.fetch_add(1, Ordering::SeqCst);
|
||||
let _permit = self.permits.acquire().await.expect("never closed");
|
||||
Reply::Done {
|
||||
|
|
@ -2717,6 +2726,7 @@ mod tests {
|
|||
model: "sonnet".to_owned(),
|
||||
effort: Some(Effort::Medium),
|
||||
max_concurrent_reviews: 2,
|
||||
max_classifications: 10,
|
||||
runtime: super::super::settings::RuntimeKind::Acp,
|
||||
acp: Some(acp.clone()),
|
||||
};
|
||||
|
|
@ -2734,6 +2744,29 @@ mod tests {
|
|||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_repo_with_more_rules_than_the_settings_allow_runs_nothing_and_posts_a_comment() {
|
||||
let mut fake = Fake::new(
|
||||
Reply::Failed {
|
||||
error: "must not run".to_owned(),
|
||||
},
|
||||
vec![crate::review_classifier::tests::rules_json_of(3)],
|
||||
);
|
||||
fake.settings.max_classifications = 2;
|
||||
let poster = Posted::default();
|
||||
run_read(&fake, &fake, &poster, &request("r")).await;
|
||||
let (_, _, review) = poster.0.into_inner().expect("unpoisoned").remove(0);
|
||||
assert_eq!(review.event, "COMMENT");
|
||||
assert!(
|
||||
review.body.contains("rules do not fit a run")
|
||||
&& review.body.contains("3 classifications")
|
||||
&& review.body.contains("1 to 2"),
|
||||
"{}",
|
||||
review.body
|
||||
);
|
||||
assert!(fake.sent.lock().expect("unpoisoned").is_empty());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn unreadable_settings_run_nothing_and_post_a_comment() {
|
||||
let fake = Fake::new(
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
//! `GET` and `PUT /api/review-classifier/settings`: the runtime, model and
|
||||
//! effort every run uses, and how many requested reviews run at once, for the
|
||||
//! whole swarm.
|
||||
//! effort every run uses, how many requested reviews run at once and how many
|
||||
//! classifications one run carries, for the whole swarm.
|
||||
//!
|
||||
//! The settings are one document under one key ([`KEY`] in [`BUCKET`]),
|
||||
//! which only this daemon reads or writes, as the schedule definitions are.
|
||||
|
|
@ -21,7 +21,9 @@ use axum::http::{HeaderMap, StatusCode};
|
|||
use problem_details::ProblemDetails;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use swarm_wire::agent_inference::Effort;
|
||||
use swarm_wire::review_classifier::{Acp, Runtime};
|
||||
use swarm_wire::review_classifier::{
|
||||
Acp, CLASSIFICATION_LIMITS, DEFAULT_MAX_CLASSIFICATIONS, Runtime,
|
||||
};
|
||||
use utoipa::ToSchema;
|
||||
|
||||
use crate::routes::{AppState, error_problem, wanted_error_status};
|
||||
|
|
@ -56,6 +58,12 @@ pub(crate) struct Settings {
|
|||
pub(crate) effort: Option<Effort>,
|
||||
/// How many requested reviews run at once, 1 to 8. Try-outs don't count.
|
||||
pub(crate) max_concurrent_reviews: usize,
|
||||
/// The most classifications one run carries, 1 to 1000: a try-out, or a
|
||||
/// review of a repo whose rules file holds more, is refused above it. The
|
||||
/// classifications of a run go one after another, each for up to three
|
||||
/// minutes. A document stored before this field has the default.
|
||||
#[serde(default = "default_max_classifications")]
|
||||
pub(crate) max_classifications: usize,
|
||||
/// What every classification runs on.
|
||||
#[serde(default)]
|
||||
pub(crate) runtime: RuntimeKind,
|
||||
|
|
@ -82,12 +90,17 @@ impl Default for Settings {
|
|||
model: swarm_wire::review_classifier::DEFAULT_MODEL.to_owned(),
|
||||
effort: None,
|
||||
max_concurrent_reviews: 2,
|
||||
max_classifications: DEFAULT_MAX_CLASSIFICATIONS,
|
||||
runtime: RuntimeKind::ApiKey,
|
||||
acp: None,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn default_max_classifications() -> usize {
|
||||
DEFAULT_MAX_CLASSIFICATIONS
|
||||
}
|
||||
|
||||
impl Settings {
|
||||
/// What is wrong with these settings, if anything.
|
||||
fn check(&self) -> Result<(), String> {
|
||||
|
|
@ -100,6 +113,14 @@ impl Settings {
|
|||
CONCURRENT_REVIEWS.end()
|
||||
));
|
||||
}
|
||||
if !CLASSIFICATION_LIMITS.contains(&self.max_classifications) {
|
||||
return Err(format!(
|
||||
"max_classifications {} is not {} to {}",
|
||||
self.max_classifications,
|
||||
CLASSIFICATION_LIMITS.start(),
|
||||
CLASSIFICATION_LIMITS.end()
|
||||
));
|
||||
}
|
||||
if let Some(acp) = &self.acp {
|
||||
check_acp(acp)?;
|
||||
}
|
||||
|
|
@ -167,6 +188,9 @@ pub(crate) struct SettingsUpdate {
|
|||
#[schema(value_type = Option<usize>)]
|
||||
max_concurrent_reviews: Option<usize>,
|
||||
#[serde(default, deserialize_with = "present")]
|
||||
#[schema(value_type = Option<usize>)]
|
||||
max_classifications: Option<usize>,
|
||||
#[serde(default, deserialize_with = "present")]
|
||||
#[schema(value_type = Option<RuntimeKind>)]
|
||||
runtime: Option<RuntimeKind>,
|
||||
#[serde(default)]
|
||||
|
|
@ -217,6 +241,9 @@ impl SettingsUpdate {
|
|||
max_concurrent_reviews: self
|
||||
.max_concurrent_reviews
|
||||
.unwrap_or(stored.max_concurrent_reviews),
|
||||
max_classifications: self
|
||||
.max_classifications
|
||||
.unwrap_or(stored.max_classifications),
|
||||
runtime: self.runtime.unwrap_or(stored.runtime),
|
||||
acp: self.acp.or(stored.acp),
|
||||
}
|
||||
|
|
@ -249,6 +276,7 @@ const FIELDS: &[&str] = &[
|
|||
"model",
|
||||
"effort",
|
||||
"max_concurrent_reviews",
|
||||
"max_classifications",
|
||||
"runtime",
|
||||
"acp",
|
||||
];
|
||||
|
|
@ -301,7 +329,7 @@ fn queue_problem(e: &anyhow::Error) -> ProblemDetails {
|
|||
responses(
|
||||
(status = 200, description = "the settings in force; with none stored, \
|
||||
`{\"model\":\"haiku\",\"effort\":null,\"max_concurrent_reviews\":2,\
|
||||
\"runtime\":\"api-key\",\"acp\":null}`",
|
||||
\"max_classifications\":10,\"runtime\":\"api-key\",\"acp\":null}`",
|
||||
body = Settings),
|
||||
(status = 503, description = "no swarm queue is configured on this host, or it is \
|
||||
not connected (problem+json)", body = String),
|
||||
|
|
@ -331,10 +359,12 @@ pub(crate) async fn get_settings(
|
|||
stored ones, which apply from the next run", body = Settings),
|
||||
(status = 400, description = "not JSON or not this shape: an unknown field, a \
|
||||
`null` other than for `effort` or `acp`, an effort other than the five \
|
||||
levels, a negative or non-integer `max_concurrent_reviews`, a runtime other \
|
||||
levels, a negative or non-integer `max_concurrent_reviews` or \
|
||||
`max_classifications`, a runtime other \
|
||||
than `api-key` or `acp`, an `acp` without `base_url`, `model` or `prices`; or, \
|
||||
once applied, a model that is empty, has whitespace or starts with `-`, \
|
||||
`max_concurrent_reviews` outside 1 to 8, an `acp.base_url` that is not \
|
||||
`max_concurrent_reviews` outside 1 to 8, `max_classifications` outside 1 to \
|
||||
1000, an `acp.base_url` that is not \
|
||||
https://, an empty `acp.model` or one with whitespace, a negative price, or \
|
||||
the `acp` runtime with no `acp` provider (problem+json)", body = String),
|
||||
(status = 401, description = "the request carries no `Remote-User`, so there is no \
|
||||
|
|
@ -419,7 +449,7 @@ mod tests {
|
|||
serde_json::to_value(Settings::default()).unwrap(),
|
||||
serde_json::json!({
|
||||
"model": "haiku", "effort": null, "max_concurrent_reviews": 2,
|
||||
"runtime": "api-key", "acp": null,
|
||||
"max_classifications": 10, "runtime": "api-key", "acp": null,
|
||||
})
|
||||
);
|
||||
assert_eq!(Settings::default().runtime(), Ok(Runtime::ApiKey));
|
||||
|
|
@ -433,6 +463,7 @@ mod tests {
|
|||
model: "sonnet".to_owned(),
|
||||
effort: Some(Effort::High),
|
||||
max_concurrent_reviews: 5,
|
||||
max_classifications: 40,
|
||||
runtime: RuntimeKind::Acp,
|
||||
acp: Some(provider()),
|
||||
};
|
||||
|
|
@ -465,6 +496,40 @@ mod tests {
|
|||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_rules_per_run_limit_is_one_to_a_thousand_and_ten_until_set() {
|
||||
for count in [0, 1001, 5000] {
|
||||
let bad = Settings {
|
||||
max_classifications: count,
|
||||
..Settings::default()
|
||||
};
|
||||
assert!(bad.check().is_err(), "{count}");
|
||||
}
|
||||
for count in [1, 10, 1000] {
|
||||
let ok = Settings {
|
||||
max_classifications: count,
|
||||
..Settings::default()
|
||||
};
|
||||
assert!(ok.check().is_ok(), "{count}");
|
||||
}
|
||||
// A document stored before the field existed decodes with the default.
|
||||
let old = r#"{"model":"haiku","effort":null,"max_concurrent_reviews":2}"#;
|
||||
assert_eq!(decode(old.as_bytes()).max_classifications, 10);
|
||||
// An update changes it alone.
|
||||
let raised = update(r#"{"max_classifications":300}"#)
|
||||
.unwrap()
|
||||
.apply(Settings::default());
|
||||
assert_eq!(
|
||||
raised,
|
||||
Settings {
|
||||
max_classifications: 300,
|
||||
..Settings::default()
|
||||
}
|
||||
);
|
||||
assert!(update(r#"{"max_classifications":null}"#).is_err());
|
||||
assert!(update(r#"{"max_classifications":-1}"#).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_bad_acp_provider_is_refused() {
|
||||
let with = |change: fn(&mut Acp)| {
|
||||
|
|
@ -571,6 +636,7 @@ mod tests {
|
|||
model: "sonnet".to_owned(),
|
||||
effort: Some(Effort::High),
|
||||
max_concurrent_reviews: 3,
|
||||
max_classifications: 25,
|
||||
runtime: RuntimeKind::Acp,
|
||||
acp: Some(provider()),
|
||||
};
|
||||
|
|
@ -666,7 +732,7 @@ mod tests {
|
|||
read,
|
||||
serde_json::json!({
|
||||
"model": "sonnet", "effort": "high", "max_concurrent_reviews": 3,
|
||||
"runtime": "api-key", "acp": null,
|
||||
"max_classifications": 10, "runtime": "api-key", "acp": null,
|
||||
})
|
||||
);
|
||||
|
||||
|
|
@ -704,7 +770,7 @@ mod tests {
|
|||
|
||||
let wanted = serde_json::json!({
|
||||
"model": "sonnet", "effort": "high", "max_concurrent_reviews": 5,
|
||||
"runtime": "api-key", "acp": null,
|
||||
"max_classifications": 10, "runtime": "api-key", "acp": null,
|
||||
});
|
||||
let (status, stored, _) = send(&app, put(&wanted.to_string())).await;
|
||||
assert_eq!(status, StatusCode::OK, "{stored}");
|
||||
|
|
@ -745,7 +811,7 @@ mod tests {
|
|||
after,
|
||||
serde_json::json!({
|
||||
"model": "sonnet", "effort": null, "max_concurrent_reviews": 4,
|
||||
"runtime": "acp", "acp": acp,
|
||||
"max_classifications": 10, "runtime": "acp", "acp": acp,
|
||||
})
|
||||
);
|
||||
let read = nats.client().await.settings().await.expect("readable");
|
||||
|
|
|
|||
|
|
@ -239,7 +239,7 @@ impl Policy {
|
|||
swarm_wire::review_classifier::SUBJECT.to_owned(),
|
||||
Self::inbox(client_id),
|
||||
]),
|
||||
responses: Some(swarm_wire::review_classifier::RUN_DEADLINE),
|
||||
responses: Some(swarm_wire::review_classifier::max_run_deadline()),
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -1669,7 +1669,7 @@ mod tests {
|
|||
#[test]
|
||||
fn the_review_classifier_hears_its_subject_and_inbox_only_and_answers_by_reply() {
|
||||
use swarm_queue_client::{agent_input_subject, client_inbox_prefix};
|
||||
use swarm_wire::review_classifier::{RUN_DEADLINE, SUBJECT};
|
||||
use swarm_wire::review_classifier::{SUBJECT, max_run_deadline};
|
||||
|
||||
let p = with_classifier()
|
||||
.permissions(CLASSIFIER)
|
||||
|
|
@ -1694,7 +1694,7 @@ mod tests {
|
|||
// It publishes nothing but replies: its own inbox keeps the list
|
||||
// non-empty, and reaches no other client.
|
||||
assert_eq!(p.publish, vec![format!("{inbox}.>")]);
|
||||
assert_eq!(p.responses, Some(RUN_DEADLINE));
|
||||
assert_eq!(p.responses, Some(max_run_deadline()));
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
|
|
|||
|
|
@ -6,6 +6,7 @@
|
|||
//! Three crates name these: the controller sends the request, the classifier
|
||||
//! answers it, and the auth-callout responder grants the one subject to each.
|
||||
|
||||
use std::ops::RangeInclusive;
|
||||
use std::time::Duration;
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
|
@ -34,18 +35,34 @@ pub const REQUEST_SUBJECT_PREFIX: &str = "$SWARM.review-request";
|
|||
/// from.
|
||||
pub const REQUEST_CONSUMER: &str = "swarm-controller";
|
||||
|
||||
/// The most classifications one run carries.
|
||||
pub const MAX_CLASSIFICATIONS: usize = 10;
|
||||
/// What the most classifications one run carries may be set to: the swarm
|
||||
/// setting's bounds. Each classification is one ask of up to
|
||||
/// [`CLASSIFICATION_DEADLINE`], run one after another, so the top of the range
|
||||
/// is a run of days; an administrator who sets it that high gets that.
|
||||
pub const CLASSIFICATION_LIMITS: RangeInclusive<usize> = 1..=1000;
|
||||
|
||||
/// The most classifications one run carries until the setting says otherwise.
|
||||
pub const DEFAULT_MAX_CLASSIFICATIONS: usize = 10;
|
||||
|
||||
/// The longest one classification may take, its first ask and any second ask
|
||||
/// together, before it is stopped and its row carries a timeout.
|
||||
pub const CLASSIFICATION_DEADLINE: Duration = Duration::from_mins(3);
|
||||
|
||||
/// The longest a whole run may take: every classification at its deadline,
|
||||
/// run one after another, plus a minute for the rest. The controller waits
|
||||
/// this long for the reply, and the queue lets the classifier answer for
|
||||
/// this long.
|
||||
pub const RUN_DEADLINE: Duration = Duration::from_mins(31);
|
||||
/// The longest a run of `classifications` may take: every classification at
|
||||
/// its deadline, run one after another, plus a minute for the rest. The
|
||||
/// controller waits this long for the reply.
|
||||
#[must_use]
|
||||
pub fn run_deadline(classifications: usize) -> Duration {
|
||||
let count = u32::try_from(classifications).unwrap_or(u32::MAX);
|
||||
CLASSIFICATION_DEADLINE.saturating_mul(count) + Duration::from_mins(1)
|
||||
}
|
||||
|
||||
/// The longest any run may take, for the biggest run the setting allows. The
|
||||
/// queue lets the classifier answer for this long.
|
||||
#[must_use]
|
||||
pub fn max_run_deadline() -> Duration {
|
||||
run_deadline(*CLASSIFICATION_LIMITS.end())
|
||||
}
|
||||
|
||||
/// The model a run uses when its request names none, and the swarm-wide
|
||||
/// setting's default.
|
||||
|
|
@ -244,8 +261,16 @@ mod tests {
|
|||
|
||||
#[test]
|
||||
fn the_run_deadline_covers_every_classification_at_its_deadline() {
|
||||
let all = CLASSIFICATION_DEADLINE * u32::try_from(MAX_CLASSIFICATIONS).expect("small");
|
||||
assert!(RUN_DEADLINE > all);
|
||||
for n in [1, DEFAULT_MAX_CLASSIFICATIONS, *CLASSIFICATION_LIMITS.end()] {
|
||||
let all = CLASSIFICATION_DEADLINE * u32::try_from(n).expect("small");
|
||||
assert!(run_deadline(n) > all, "{n} classifications");
|
||||
}
|
||||
// A default-sized run waits the half hour it always has.
|
||||
assert_eq!(
|
||||
run_deadline(DEFAULT_MAX_CLASSIFICATIONS),
|
||||
Duration::from_mins(31)
|
||||
);
|
||||
assert_eq!(max_run_deadline(), run_deadline(1000));
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
|
|
|||
Loading…
Reference in a new issue