From c0af8d1fd335a13a0b15d7fbec973bb6b0bac524 Mon Sep 17 00:00:00 2001 From: atlas Date: Sat, 10 Oct 2026 18:05:14 +0200 Subject: [PATCH] docs: review classifier architecture (planned) Add docs/swarm/review-classifier.md, the high-level architecture of the planned PR review classifier pipeline, written as the full target shape under a "planned, not implemented yet" banner. Link it from docs/README.md and the docs/swarm/README.md piece list. The page reflects the epic description and the operator's answers in the epic thread; points still open are listed in its "Still open" section. The banner names the epic by title rather than by number because the tracker-tag lint rejects issue references in tracked files. --- docs/README.md | 3 + docs/swarm/README.md | 3 + docs/swarm/review-classifier.md | 295 ++++++++++++++++++++++++++++++++ 3 files changed, 301 insertions(+) create mode 100644 docs/swarm/review-classifier.md diff --git a/docs/README.md b/docs/README.md index f5d93e54..70e05794 100644 --- a/docs/README.md +++ b/docs/README.md @@ -86,6 +86,9 @@ declarations. twice after downtime?** → [`swarm/schedules.md`](swarm/schedules.md). - **How do I set an agent's model or effort from the swarm, and what happens to its own pick?** → [`swarm/inference.md`](swarm/inference.md). +- **How will a PR get checked against its repo's own yes/no rules before + argus reviews it?** → [`swarm/review-classifier.md`](swarm/review-classifier.md) + (planned, not implemented yet). ## Scheduler, CI, observability diff --git a/docs/swarm/README.md b/docs/swarm/README.md index 03fb6f1d..e7bac767 100644 --- a/docs/swarm/README.md +++ b/docs/swarm/README.md @@ -25,6 +25,9 @@ grafana, the controller, authelia, …) → kept by the swarm and fired in each agent's container. → [`schedules.md`](schedules.md) - **inference settings** — the model and effort the swarm sets for an agent, over its own. → [`inference.md`](inference.md) +- **review classifier** — planned, not implemented yet: checks a PR against + its repo's own yes/no rules and posts the result as a review. → + [`review-classifier.md`](review-classifier.md) - **swarm UI** — the operator's day-to-day surface, on the swarm apex, `admins` only. → [`ui.md`](ui.md) - **shared services** — one forge, homeserver, SSO, queue and metrics/logs diff --git a/docs/swarm/review-classifier.md b/docs/swarm/review-classifier.md new file mode 100644 index 00000000..2caad8ca --- /dev/null +++ b/docs/swarm/review-classifier.md @@ -0,0 +1,295 @@ +# Review classifier + +> ⚠️ **PLANNED, NOT IMPLEMENTED YET.** This page describes the full target +> shape of the review classifier at component level, and none of it exists +> in the tree today. The forge epic "pr review classifier pipeline" tracks +> the work. Details land here as each part ships, and this banner goes away +> when the epic closes. + +The review classifier checks a pull request against short, named yes/no +rules that the repo itself defines, and posts the result as a forge review. +A small model answers one rule at a time, so mundane rule checks come out +the same way on every PR. + +It doesn't replace argus's review. argus reviews a PR only once the +classifier has passed it, and argus runs on a bigger model. The classifier +takes the cheap, repetitive checks off argus. + +## Repo configuration + +A repo opts in with a config file in a `.hyperhive/` directory at its root, +in YAML or JSON. The classifier always reads it from the repo's **default +branch**, never from the PR, so a PR can't edit the rules it's judged by. + +Each classification has: + +- a **name**, short and unique in the repo, shown in the result table +- a **description**: the yes/no question the model answers about the PR +- a **certainty threshold** from 0 to 100 +- a **polarity**: _positive_ means a match is the wanted outcome, and + _negative_ means a match is a violation + +An illustrative example. The file name and every field name in it are +placeholders, not a fixed schema: + +```yaml +# illustrative only: file name and field names aren't fixed +classifications: + - name: datetime + description: All datetimes carry time zone information. + polarity: positive + threshold: 80 + - name: no-ads + description: >- + A commit message contains a co-authored-by line or other + advertising for an AI model. + polarity: negative + threshold: 90 + - name: bao-first + description: Any added secret reaches its reader through the bao store. + polarity: positive + threshold: 70 + - name: docs-updated + description: >- + The PR updates the operator-facing docs where the code change + needs it. + polarity: positive + threshold: 60 +``` + +## Trigger and lifecycle + +The classifier has a dedicated forge service account. Requesting that +account as a reviewer on a PR starts a run: + +1. The forge sends the review-request webhook to + [`swarm-controller`](../../swarm-controller/README.md), the swarm's one + webhook owner. +2. The controller hands the run to the classifier service as an async + message. +3. The classifier runs every classification in the repo's config, read from + the default branch, against the PR's commits and diff. +4. The service account posts the result on the PR as a review. + +A verdict belongs to the commit it ran on. Once the PR's head moves, +requesting the service account as a reviewer again starts a fresh run on the +new head. + +That only holds if the bot's review on an older commit stops counting once +the head moves. Either the repo enables Forgejo's stale-review dismissal +(see [PR review gate](../process/pr-review-gate.md)), or the service +dismisses its own earlier review when it posts a new one. Where neither +applies, an old approval keeps counting and an old request-changes keeps +blocking. + +Runs go in parallel, up to a maximum set at swarm level. + +## Components + +### The classifier service + +A swarm-level service in its own container, on whichever host the operator +puts it. It talks to `swarm-controller` through async messages only, and +gets its secrets from the swarm's secret store, bao +([`bao.md`](bao.md), [`credentials.md`](credentials.md)). + +It looks like an agent from the inside, with three things missing: it has no +full agent identity, no state directory and no tools. Every model session is +ephemeral, and nothing carries over from one run to the next. + +The model sessions run through [`hive-runtime`](../../hive-runtime/README.md), +the crate `hive-agent` and `hive-subagent-mcp` already drive their turns +through: the `Runtime` trait in `hive-runtime/src/lib.rs`, with +`ClaudeRuntime` and `AcpRuntime` behind it. That trait runs turns on a durable +session, and the subagent daemon's claude path +(`hive-subagent-mcp/src/session.rs`) spawns claude directly. A refactor +gives `hive-runtime` a one-off, cancellable session shape first, so the +classifier, the agent and the subagent daemon share one runtime +implementation. + +### Swarm-level settings + +The operator sets the classifier's settings for the whole swarm: + +- the runtime (see [Runtimes](#runtimes)) and its credentials +- the model and effort, and related inference settings +- the maximum number of concurrent runs + +The per-agent equivalent is [`inference.md`](inference.md), where the +controller writes model and effort to a queue KV bucket +(`swarm-queue-client/src/agent_inference.rs`) and each agent reads its own +key. + +### The service account + +A dedicated forge user, separate from swarm-controller's site-admin account +and from every agent's account. It's the reviewer someone requests, it +authors every classifier review, and it pushes the branches the try-out page +creates. + +## The classification run + +### Shared context + +Every classification in a run sees the same context, in the same order: + +1. a short system prompt +2. the PR's commits, each with its full message +3. the full diff + +Only the last part differs per classification: that classification's +description and the task to classify the PR against it. The identical prefix +lets the runtime's prompt cache reuse it across questions. + +### One question, one answer + +The model answers each classification in JSON. The answer holds: + +- a **verdict**, one of _match_, _no match_ or _not applicable to this PR_ +- a **certainty** from 0 to 100, how sure the model is of that verdict +- a **reason**, limited in length + +An illustrative answer, with placeholder field names: + +```json +{ + "verdict": "no-match", + "certainty": 92, + "reason": "Two new timestamps use naive datetimes without a zone." +} +``` + +### Aggregation into the verdict + +The classifier collects the answers into one result, keyed by +classification name, and logs it. A classification counts toward the +overall verdict only if it's applicable and its certainty reaches the +threshold. The table still lists every classification below its threshold or +not applicable, marked as excluded. + +A counted classification fails when its verdict lands on the wrong side of +its polarity: _no match_ on a positive one, _match_ on a negative one. Any +failing classification makes the verdict **request changes**. Otherwise the +verdict is **approve**, including when the run counted no classification at +all. + +### The review + +The service account posts the verdict as a forge review, approve or request +changes, and the table is the review's body. An illustrative table: + +| classification | polarity | verdict | certainty | threshold | counted | reason | +| -------------- | -------- | -------------- | --------- | --------- | ------- | -------------------------------------------- | +| datetime | positive | no match | 92 | 80 | yes ❌ | Two new timestamps use naive datetimes. | +| no-ads | negative | no match | 99 | 90 | yes ✅ | No commit message carries an AI attribution. | +| bao-first | positive | not applicable | 95 | 70 | no | The PR adds no secret. | +| docs-updated | positive | match | 41 | 60 | no | Below threshold. | + +**Overall: request changes.** + +The approval is a real one: it counts toward a repo's required approvals +like any other ([`pr-review-gate.md`](../process/pr-review-gate.md)). The +hyperhive repo requires more approvals to account for it. + +## Runtimes + +The classifier runs on one of three runtimes, chosen at swarm level: + +- **claude subscription**: the `claude` CLI on a Claude subscription login +- **API key**: the `claude` CLI on an API key +- **ACP**: an ACP agent and its provider, as for an ACP agent today + +Model names don't carry across runtimes. A claude alias means nothing to an +ACP provider, so the model setting belongs to the runtime it names. + +> **Open point.** The subscription runtime needs a login, which is the one +> piece of durable state the service would keep. A swarm-level claude +> subscription login may have to exist before this runtime works. + +## Swarm UI + +Two pages in the [swarm UI](ui.md), `admins` only like the rest of it. + +### Recent runs + +One card per run: the repo, the PR, how many classifications ran, and the +run's overall cost. Selecting a card expands it to show each +classification's result, the same rows as the review table. + +### Try-out + +A page for writing classification prompts without touching the repo. Pick a +repo and a PR, write or edit a classification, and run it against that PR; +change the prompt and run it again until it answers the way you want. +Nothing reaches the forge while you iterate. + +When it's ready, **open a PR** has the service account push a branch that +adds the classification to the repo's config, with you, the swarm UI user, +as commit author. The page then links you to the forge's new-PR page for +that branch, where you log in as usual and open the PR yourself. + +## Observability and cost + +Every run records its cost, which the recent-runs page shows. Where the cost +comes from depends on the runtime: + +- An ACP agent reports a running cost total (`ReportedCost` in + `hive-runtime/src/acp/stream.rs`); a session's last total is its cost. +- The `claude` CLI reports token counts, including cache reads and writes. + The dashboard already estimates a cost from token counts with a price + table (`modelPrices` in [`dashboard.md`](../web-ui/dashboard.md)). +- On a subscription, cost is notional. + +The cache token counts also show whether the shared context gets the cache +reuse it's meant to. + +The classifier logs each run's result to its journal, which ships to +VictoriaLogs like every swarm service container's +([`services.md`](services.md#logs-victorialogs)). + +## Still open + +These get settled, and filled in here, as the work reaches them: + +- **Cache strategy.** Fully independent sessions per classification, or one + warm session and then the rest in parallel. To measure through the cache + token counts before choosing. +- **Run store and retention.** Where run records live, how long they stay, + and how the recent-runs page reads them. +- **Failure handling.** What a runtime error, malformed JSON, a timeout or a + rate limit does to a classification and to the verdict. +- **JSON validation and the reason cap.** The exact answer schema, and the + reason's length limit. +- **Oversized diffs.** What happens when the diff doesn't fit the model's + context, and whether lockfiles collapse as in `hive-forge diff` + (`hive-forge/src/verbs/diff.rs`). +- **Stale reviews.** Whether the repo's stale-review dismissal setting + retires the bot's review on an older commit, or the service dismisses its + own earlier review when it posts a new one. +- **Budget.** Any per-run or per-day spend limit on top of the concurrency + cap. +- **Diff-leaves-swarm policy.** On the API key and ACP runtimes the diff + goes to a third-party provider. Which repos may use which runtime, and + whether anything gets redacted. +- **Prompt injection.** A PR's diff and commit messages are model input, so + a PR can try to talk the classifier into approving it. +- **The "no tools" mechanism.** How each runtime runs a session with no + tools at all. +- **Credential paths.** Where in bao the runtime credentials and the service + account's forge token live, and which component holds the token. +- **Model naming across runtimes.** How the swarm-level model setting + handles runtime-specific model names. +- **Cost method per runtime and telemetry labels.** The exact cost source + per runtime, and the resource attributes that keep the classifier's spend + separate from the agents' on the existing dashboards. +- **Async message medium.** Which queue subjects or streams carry the + messages between `swarm-controller` and the classifier. +- **argus handoff.** How argus learns the classifier passed a PR before it + starts its own review. +- **Opt-in details.** Beyond the config file: the service account's access + to a repo, so someone can request it as reviewer and it can push try-out + branches. +- **Commit author for try-out PRs.** How the classifier derives the commit + author's name and email from the swarm UI login. +- **Refactor scope.** How much of `hive-agent`'s and `hive-subagent-mcp`'s + runtime code moves into `hive-runtime` before the classifier uses it.