vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity
hyperhive/docs/swarm/review-classifier.md
atlas 2b1573c2ba swarm-controller: read review rules at a literal nested path 
forgejo-api escapes the whole file path of a raw-file request, so the
rules file went out as `raw/.hyperhive%2Freview-classifier.yml` and
reached the forge only if every proxy and router in between decoded the
`%2F`. The controller now sends that one request itself: `raw_file_url`
escapes each path segment on its own and joins them with `/`, and
`forge::Client` holds a second HTTP client with the same token for it.

A 404 from the default-branch lookup now means "no commit on the
default branch", like an empty repo, instead of falling through to the
route's "no repo" answer. Only the repo lookup's 404 still means the
repo doesn't exist.

Tests cover the URL shape, the reads against a loopback stub forge
that serves only the literal path, the branch 404, 503 with no forge
access, 503 at the forge deadline on a paused clock, and a bad
polarity value.

Part of #5096.
2026-10-12 14:54:59 +02:00

334 lines
15 KiB
Markdown
Raw Blame History

# Review classifier
> ⚠️ **PLANNED, NOT IMPLEMENTED YET.** This page describes the full target
> shape of the review classifier at component level, and none of it exists
> in the tree today. The forge epic "pr review classifier pipeline" tracks
> the work. Details land here as each part ships, and this banner goes away
> when the epic closes.
The review classifier checks a pull request against short, named yes/no
rules that the repo itself defines, and posts the result as a forge review.
A small model answers one rule at a time, so mundane rule checks come out
the same way on every PR.
It doesn't replace a full review by a human or agent reviewer. A swarm can
request that review only once the classifier has passed the PR, and use a
larger model there. The classifier takes the cheap, repetitive checks off
that reviewer.
## Repo configuration
A repo opts in with a rules file at its root, in YAML as
`.hyperhive/review-classifier.yml` or in JSON as
`.hyperhive/review-classifier.json`. It holds one of the two; with both, the
rules don't load. `swarm-controller` always reads the file from the repo's
**default branch**, never from the PR, so a PR can't edit the rules it's
judged by. It reads it with its own forge token on every load, with no cache,
so a rule merged to the default branch counts from the next load on.
The file holds one key, `classifications`: a list of one to ten entries, each
with exactly these four keys:
- `name`: short and unique in the repo, shown in the result table
- `description`: the yes/no question the model answers about the PR
- `threshold`: the certainty, 0 to 100, a verdict needs to count
- `polarity`: `positive` means a match is the wanted outcome, and
`negative` means a match is a violation
Any other key, at the top or in an entry, is an error: a misspelt key fails
the load instead of vanishing. An empty list, more than ten entries, an empty
name or description, and two entries with one name are errors too. Nothing
falls back to a pass: a repo whose file doesn't load has no rules.
An example:
```yaml
classifications:
- name: datetime
description: All datetimes carry time zone information.
polarity: positive
threshold: 80
- name: no-ads
description: >-
A commit message contains a co-authored-by line or other
advertising for an AI model.
polarity: negative
threshold: 90
- name: bao-first
description: Any added secret reaches its reader through the bao store.
polarity: positive
threshold: 70
- name: docs-updated
description: >-
The PR updates the operator-facing docs where the code change
needs it.
polarity: positive
threshold: 60
```
## Trigger and lifecycle
The classifier has a dedicated forge service account. Requesting that
account as a reviewer on a PR starts a run:
1. The forge sends the review-request webhook to
[`swarm-controller`](../../swarm-controller/README.md), the swarm's one
webhook owner.
2. The controller hands the run to the classifier service as an async
message.
3. The classifier runs every classification in the repo's config, read from
the default branch, against the PR's commits and diff.
4. The service account posts the result on the PR as a review.
A verdict belongs to the commit it ran on. Once the PR's head moves,
requesting the service account as a reviewer again starts a fresh run on the
new head.
That only holds if the bot's review on an older commit stops counting once
the head moves. Either the repo enables Forgejo's stale-review dismissal
(see [PR review gate](../process/pr-review-gate.md)), or the service
dismisses its own earlier review when it posts a new one. Where neither
applies, an old approval keeps counting and an old request-changes keeps
blocking.
Runs go in parallel, up to a maximum set at swarm level.
## Components
### The classifier service
A swarm-level service in its own container, on whichever host the operator
puts it. It talks to `swarm-controller` through async messages only, and
gets its secrets from the swarm's secret store, bao
([`bao.md`](bao.md), [`credentials.md`](credentials.md)).
It looks like an agent from the inside, with three things missing: it has no
full agent identity, no state directory and no tools. Every model session is
ephemeral, and nothing carries over from one run to the next.
On the claude runtimes, each classification is one `claude --print` the
service spawns through the `hive-claude` crate directly, with no session to
resume, no tools (`--tools ""`), no MCP server (`--strict-mcp-config` and no
config) and an empty directory of its own. A classification needs no durable
session, so the service doesn't use [`hive-runtime`](../../hive-runtime/README.md),
whose `Runtime` trait drives the agents' durable sessions.
The service is `swarm-review-classifier`, and its container is the one
[`services.md`](services.md#review-classifier) describes. It holds no forge
credential: `swarm-controller` reads the PR and sends it along.
### Swarm-level settings
The operator sets the classifier's settings for the whole swarm:
- the runtime (see [Runtimes](#runtimes)) and its credentials
- the model and effort, and related inference settings
- the maximum number of concurrent runs
The per-agent equivalent is [`inference.md`](inference.md), where the
controller writes model and effort to a queue KV bucket
(`swarm-queue-client/src/agent_inference.rs`) and each agent reads its own
key.
### The service account
A dedicated forge user, separate from swarm-controller's site-admin account
and from every agent's account. It's the reviewer someone requests, it
authors every classifier review, and it pushes the branches the try-out page
creates.
## The classification run
### Shared context
Every classification in a run sees the same context, in the same order:
1. a short system prompt
2. the PR's commits, each with its full message
3. the full diff
Only the last part differs per classification: that classification's
description and the task to classify the PR against it. The identical prefix
lets the runtime's prompt cache reuse it across questions.
### One question, one answer
The model answers each classification in JSON. The answer holds:
- a **verdict**, one of _match_, _no match_ or _not applicable to this PR_
- a **certainty** from 0 to 100, how sure the model is of that verdict
- a **reason**, at most 500 characters
An answer:
```json
{
"verdict": "no-match",
"certainty": 92,
"reason": "Two new timestamps use naive datetimes without a zone."
}
```
The verdict field holds `match`, `no-match` or `not-applicable`, and the
certainty field a whole number. A runtime error, a timeout, or an answer that
isn't that JSON object leaves the classification without a verdict: it's listed
with its error, and doesn't count. A run with no API key, or in which no
classification got an answer, fails as a whole, and shows its error instead of a
verdict.
### Aggregation into the verdict
The classifier collects the answers into one result, keyed by
classification name, and logs it. A classification counts toward the
overall verdict only if it's applicable and its certainty reaches the
threshold. The table still lists every classification below its threshold or
not applicable, marked as excluded.
A counted classification fails when its verdict lands on the wrong side of
its polarity: _no match_ on a positive one, _match_ on a negative one. Any
failing classification makes the verdict **request changes**. Otherwise the
verdict is **approve**, including when the run counted no classification at
all.
### The review
The service account posts the verdict as a forge review, approve or request
changes, and the table is the review's body. An illustrative table:
| classification | polarity | verdict | certainty | threshold | counted | reason |
| -------------- | -------- | -------------- | --------- | --------- | ------- | -------------------------------------------- |
| datetime | positive | no match | 92 | 80 | yes ❌ | Two new timestamps use naive datetimes. |
| no-ads | negative | no match | 99 | 90 | yes ✅ | No commit message carries an AI attribution. |
| bao-first | positive | not applicable | 95 | 70 | no | The PR adds no secret. |
| docs-updated | positive | match | 41 | 60 | no | Below threshold. |
**Overall: request changes.**
The approval is a real one: it counts toward a repo's required approvals
like any other ([`pr-review-gate.md`](../process/pr-review-gate.md)). The
Repos that rely on it typically raise their required approval count to
account for it.
## Runtimes
The classifier runs on one of three runtimes, chosen at swarm level:
- **claude subscription**: the `claude` CLI on a Claude subscription login
- **API key**: the `claude` CLI on an API key
- **ACP**: an ACP agent and its provider, as for an ACP agent today
Model names don't carry across runtimes. A claude alias means nothing to an
ACP provider, so the model setting belongs to the runtime it names.
> **Open point.** The subscription runtime needs a login, which is the one
> piece of durable state the service would keep. A swarm-level claude
> subscription login may have to exist before this runtime works.
## Swarm UI
Two pages in the [swarm UI](ui.md), `admins` only like the rest of it.
### Recent runs
One card per run: the repo, the PR, and how many classifications ran.
Selecting a card expands it to show each classification's result, the same
rows as the review table.
### Try-out
A page for writing classification prompts without touching the repo. Pick a
repo and a PR, write or edit a classification, and run it against that PR;
change the prompt and run it again until it answers the way you want.
Nothing reaches the forge while you iterate.
The page posts the run to `swarm-controller` (`POST
/api/review-classifier/try-out`), which reads the PR with its own forge token
and sends it to the classifier as one queue request on
`$SWARM.review-classifier.try-out`. The page then reads the result back (`GET
/api/review-classifier/try-out/{id}`): a run can take longer than a request
stays open. The controller keeps the result in memory for an hour; there is no
run store.
`GET /api/review-classifier/rules?repo=owner/name` returns the repo's own
rules, for the page to load into its editor; a run posts them like typed
ones. The route answers:
- **200** with `path` (the file read), `branch` (the default branch),
`commit` (the commit the route read the file at) and `classifications`, in
the file's order and in the POST's shape
- **400** for a `repo` that's missing or not `owner/name`
- **404** for a repo the forge doesn't know; a default branch with no commit,
in an empty repo or when the default branch doesn't exist; or a default
branch with neither file
- **422** for both files on the default branch, or a file that doesn't parse
or breaks a rule from [Repo configuration](#repo-configuration); the
`detail` names the file and what's wrong
- **503** when this host has no forge token yet, or the forge failed or
didn't answer within two minutes
Every error is problem+json with the reason in `detail`.
When it's ready, **open a PR** has the service account push a branch that
adds the classification to the repo's config, with you, the swarm UI user,
as commit author. The page then links you to the forge's new-PR page for
that branch, where you log in as usual and open the PR yourself.
## Observability and cost
The swarm UI shows no token counts, cache counts or cost, on any page. On the
claude runtimes, every classifier run starts its `claude` process with
`OTEL_RESOURCE_ATTRIBUTES` naming the repo and the PR, so the cost reads from
the metrics store by repo and PR. The process exports to the hive-tier collector on
the classifier's host, where that host runs one. On a subscription, cost is notional.
On the ACP runtime, the harness counts the agent's `usage_update` running
totals and exports them as `hyperhive.agent.cost.usage`, labelled with only
`model` and `currency`
([`observability.md`](../scheduler/observability.md#acp-reported-cost)). That
cost has no repo or PR label yet.
The classifier logs each run's result to its journal, which ships to
VictoriaLogs like every swarm service container's
([`services.md`](services.md#logs-victorialogs)).
## Still open
These get settled, and filled in here, as the work reaches them:
- **Cache strategy.** Fully independent sessions per classification, or one
warm session and then the rest in parallel. To measure through the cache
token counts before choosing.
- **Run store and retention.** Where run records live, how long they stay,
and how the recent-runs page reads them.
- **Oversized diffs.** What happens when the diff doesn't fit the model's
context. Lockfiles already collapse to one line each, as in `hive-forge diff`.
- **Stale reviews.** Whether the repo's stale-review dismissal setting
retires the bot's review on an older commit, or the service dismisses its
own earlier review when it posts a new one.
- **Budget.** Any per-run or per-day spend limit on top of the concurrency
cap.
- **Diff-leaves-swarm policy.** On the API key and ACP runtimes the diff
goes to a third-party provider. Which repos may use which runtime, and
whether anything gets redacted.
- **Prompt injection.** A PR's diff and commit messages are model input, so
a PR can try to talk the classifier into approving it.
- **The "no tools" mechanism on ACP.** How the ACP runtime runs a session with
no tools at all. The claude runtimes pass `--tools ""`.
- **Credential paths.** Where in bao the ACP and subscription runtimes'
credentials and the service account's forge token live, and which component
holds the token. The API key is at
`secret/swarm/services/review-classifier/anthropic-api-key`, field `api_key`.
- **Model naming across runtimes.** How the swarm-level model setting
handles runtime-specific model names.
- **ACP cost labels.** How an ACP run's cost gets a repo and PR label, so
it reads from the metrics store like the claude runtimes' cost.
- **Async message medium for review runs.** Which queue subjects or streams
carry a webhook-started run. A try-out is one request on
`$SWARM.review-classifier.try-out`, answered on its reply subject.
- **Reviewer handoff.** How a later reviewer, human or agent, learns the
classifier passed a PR before it starts its own review.
- **Opt-in details.** Beyond the config file: the service account's access
to a repo, so someone can request it as reviewer and it can push try-out
branches.
- **Commit author for try-out PRs.** How the classifier derives the commit
author's name and email from the swarm UI login.
View git blame Copy permalink