hyperhive/docs/swarm/review-classifier.md

Review classifier

⚠️ PLANNED, NOT IMPLEMENTED YET. This page describes the full target shape of the review classifier at component level, and none of it exists in the tree today. The forge epic "pr review classifier pipeline" tracks the work. Details land here as each part ships, and this banner goes away when the epic closes.

The review classifier checks a pull request against short, named yes/no rules that the repo itself defines, and posts the result as a forge review. A small model answers one rule at a time, so mundane rule checks come out the same way on every PR.

It doesn't replace a full review by a human or agent reviewer. A swarm can request that review only once the classifier has passed the PR, and use a larger model there. The classifier takes the cheap, repetitive checks off that reviewer.

Repo configuration

A repo opts in with a rules file at its root, in YAML as .hyperhive/review-classifier.yml or in JSON as .hyperhive/review-classifier.json. It holds one of the two; with both, the rules don't load. swarm-controller always reads the file from the repo's default branch, never from the PR, so a PR can't edit the rules it's judged by. It reads it with its own forge token on every load, with no cache, so a rule merged to the default branch counts from the next load on.

The file holds one key, classifications: a list of one to ten entries, each with exactly these four keys:

• name: short and unique in the repo, shown in the result table
• description: the yes/no question the model answers about the PR
• threshold: the certainty, 0 to 100, a verdict needs to count
• polarity: positive means a match is the wanted outcome, and negative means a match is a violation

Any other key, at the top or in an entry, is an error: a misspelt key fails the load instead of vanishing. An empty list, more than ten entries, an empty name or description, and two entries with one name are errors too. Nothing falls back to a pass: a repo whose file doesn't load has no rules.

An example:

classifications:
  - name: datetime
    description: All datetimes carry time zone information.
    polarity: positive
    threshold: 80
  - name: no-ads
    description: >-
      A commit message contains a co-authored-by line or other
      advertising for an AI model.
    polarity: negative
    threshold: 90
  - name: bao-first
    description: Any added secret reaches its reader through the bao store.
    polarity: positive
    threshold: 70
  - name: docs-updated
    description: >-
      The PR updates the operator-facing docs where the code change
      needs it.
    polarity: positive
    threshold: 60

Trigger and lifecycle

The classifier has a dedicated forge service account, review-classifier (see The service account). Requesting that account as a reviewer on a PR starts a run:

1. The forge sends the pull_request_review_request delivery to swarm-controller, the swarm's one webhook owner, on its instance-wide hook /webhook/forge/review-request. A delivery for any other reviewer, or a removed request, does nothing.
2. The controller reads the PR and the repo's rules from the default branch, and hands the run to the classifier service as one queue request on $SWARM.review-classifier.try-out, the subject a try-out uses.
3. The classifier runs every classification against the PR's commits and diff.
4. The controller posts the result on the PR as the service account's review, on the commit it read.

A verdict belongs to the commit it ran on. A push starts no run. Once the PR's head moves, requesting the service account as a reviewer again starts a fresh run on the new head.

The re-request also retires the account's earlier review. The forge clears the official flag on a reviewer's earlier reviews when it's requested again, and only official reviews count toward the required approvals or block a merge. Removing the request before the new review lands restores the earlier one. A repo whose branch protection also dismisses stale approvals (see PR review gate) stops counting an approval on an older commit even without a re-request.

At most two requested reviews run at once, and up to eight more wait in the controller's memory, so a controller restart drops them, and nothing gets posted for them. A request for a PR and head that's already running or waiting starts nothing. A request with all ten places taken starts no run: the account posts a **No verdict:** comment saying so.

Components

The classifier service

A swarm-level service in its own container, on whichever host the operator puts it. It talks to swarm-controller through async messages only, and gets its secrets from the swarm's secret store, bao (bao.md, credentials.md).

It looks like an agent from the inside, with three things missing: it has no full agent identity, no state directory and no tools. Every model session is ephemeral, and nothing carries over from one run to the next.

On the claude runtimes, each classification is one claude --print the service spawns through the hive-claude crate directly, with no session to resume, no tools (--tools ""), no MCP server (--strict-mcp-config and no config) and an empty directory of its own. A classification needs no durable session, so the service doesn't use hive-runtime, whose Runtime trait drives the agents' durable sessions.

The service is swarm-review-classifier, and its container is the one services.md describes. It holds no forge credential: swarm-controller reads the PR and sends it along.

Swarm-level settings

The operator sets the classifier's settings for the whole swarm:

• the runtime (see Runtimes) and its credentials
• the model and effort, and related inference settings
• the maximum number of concurrent runs

The per-agent equivalent is inference.md, where the controller writes model and effort to a queue KV bucket (swarm-queue-client/src/agent_inference.rs) and each agent reads its own key.

The service account

A dedicated forge user, separate from swarm-controller's site-admin account and from every agent's account. It's the reviewer someone requests, it authors every classifier review, and it pushes the branches the try-out page creates.

swarm-controller creates it as review-classifier, with the email review-classifier@services.hyperhive.local and no repo creation, and mints its one token, swarm-review-classifier with scope write:repository, into the store at swarm/services/review-classifier/forge/token. A pass at start and every five minutes re-mints when the token is missing or no longer matches the forge. The controller refuses an existing account by that name with any other email. The controller posts the reviews with that token, and the classifier service never sees it.

Giving a repo access is a repo admin's step, by hand. Nothing grants it automatically. In the repo's settings, add review-classifier as a collaborator:

• with read, so someone can request it as a reviewer, and its reviews show, but its approval doesn't count
• with write, so its approval counts toward the required approvals and its request-changes blocks the merge where the branch protection blocks on rejected reviews. Where the branch protection restricts approvals to an allowlist, the account counts only once it's on that list, and read access is then enough.

The repo also needs a rules file on its default branch (see Repo configuration). Without one, a request posts a comment saying so.

The classification run

Shared context

Every classification in a run sees the same context, in the same order:

1. a short system prompt
2. the PR's commits, each with its full message
3. the full diff

Only the last part differs per classification: that classification's description and the task to classify the PR against it. The identical prefix lets the runtime's prompt cache reuse it across questions.

One question, one answer

The model answers each classification in JSON. The answer holds:

• a verdict, one of match, no match or not applicable to this PR
• a certainty from 0 to 100, how sure the model is of that verdict
• a reason, at most 500 characters

An answer:

{
  "verdict": "no-match",
  "certainty": 92,
  "reason": "Two new timestamps use naive datetimes without a zone."
}

The verdict field holds match, no-match or not-applicable, and the certainty field a whole number. A runtime error, a timeout, or an answer that isn't that JSON object leaves the classification without a verdict: it's listed with its error, and doesn't count. A run with no API key, or in which no classification got an answer, fails as a whole, and shows its error instead of a verdict.

Aggregation into the verdict

The classifier collects the answers into one result, keyed by classification name, and logs it. A classification counts toward the overall verdict only if it's applicable and its certainty reaches the threshold. The table still lists every classification below its threshold or not applicable, marked as excluded.

A counted classification fails when its verdict lands on the wrong side of its polarity: no match on a positive one, match on a negative one. Any failing classification makes the verdict request changes. Otherwise the verdict is approve, including when the run counted no classification at all.

The review

The service account posts the verdict as a forge review, approve or request changes, and the table is the review's body. An illustrative table:

classification	polarity	verdict	certainty	threshold	counted	reason
datetime	positive	no match	92	80	yes ❌	Two new timestamps use naive datetimes.
no-ads	negative	no match	99	90	yes ✅	No commit message carries an AI attribution.
bao-first	positive	not applicable	95	70	no	The PR adds no secret.
docs-updated	positive	match	41	60	no	Below threshold.

Overall: request changes.

The approval is a real one: it counts toward a repo's required approvals like any other (pr-review-gate.md). The Repos that rely on it typically raise their required approval count to account for it.

A run with no clean verdict never approves. When the PR or the rules don't load, the run fails, the classifier answers a different number of classifications than the controller sent it, or any classification has no answer, the account posts a comment instead, starting with **No verdict:** and the reason, followed by the table when there is one. A comment neither approves nor blocks the merge: nine timeouts and one pass post a comment, not an approval.

The controller flattens every cell to one line, with |, backticks, brackets, <, >, &, # and other markup characters escaped, and a zero-width space after each @, #, !, :// and www., so a model's reason can't break the table, embed markup, mention anyone, reference an issue or PR, or carry a link.

Runtimes

The classifier runs on one of three runtimes, chosen at swarm level:

• claude subscription: the claude CLI on a Claude subscription login
• API key: the claude CLI on an API key
• ACP: an ACP agent and its provider, as for an ACP agent today

Model names don't carry across runtimes. A claude alias means nothing to an ACP provider, so the model setting belongs to the runtime it names.

Open point. The subscription runtime needs a login, which is the one piece of durable state the service would keep. A swarm-level claude subscription login may have to exist before this runtime works.

Swarm UI

Two pages in the swarm UI, admins only like the rest of it.

Recent runs

One card per run: the repo, the PR, and how many classifications ran. Selecting a card expands it to show each classification's result, the same rows as the review table.

Try-out

A page for writing classification prompts without touching the repo. Pick a repo and a PR, write or edit a classification, and run it against that PR; change the prompt and run it again until it answers the way you want. Nothing reaches the forge while you iterate.

The page posts the run to swarm-controller (POST /api/review-classifier/try-out), which reads the PR with its own forge token and sends it to the classifier as one queue request on $SWARM.review-classifier.try-out. The page then reads the result back (GET /api/review-classifier/try-out/{id}): a run can take longer than a request stays open. The controller keeps the result in memory for an hour; there is no run store.

GET /api/review-classifier/rules?repo=owner/name returns the repo's own rules, for the page to load into its editor; a run posts them like typed ones. The route answers:

• 200 with path (the file read), branch (the default branch), commit (the commit the route read the file at) and classifications, in the file's order and in the POST's shape
• 400 for a repo that's missing or not owner/name
• 404 for a repo the forge doesn't know; a default branch with no commit, in an empty repo or when the default branch doesn't exist; or a default branch with neither file
• 422 for both files on the default branch, or a file that doesn't parse or breaks a rule from Repo configuration; the detail names the file and what's wrong
• 503 when this host has no forge token yet, or the forge failed or didn't answer within two minutes

Every error is problem+json with the reason in detail.

When it's ready, open a PR has the service account push a branch that adds the classification to the repo's config, with you, the swarm UI user, as commit author. The page then links you to the forge's new-PR page for that branch, where you log in as usual and open the PR yourself.

Observability and cost

The swarm UI shows no token counts, cache counts or cost, on any page. On the claude runtimes, every classifier run starts its claude process with OTEL_RESOURCE_ATTRIBUTES naming the repo and the PR, so the cost reads from the metrics store by repo and PR. The process exports to the hive-tier collector on the classifier's host, where that host runs one. On a subscription, cost is notional.

On the ACP runtime, the harness counts the agent's usage_update running totals and exports them as hyperhive.agent.cost.usage, labelled with only model and currency (observability.md). That cost has no repo or PR label yet.

The classifier logs each run's result to its journal, which ships to VictoriaLogs like every swarm service container's (services.md).

Still open

These get settled, and filled in here, as the work reaches them:

• Cache strategy. Fully independent sessions per classification, or one warm session and then the rest in parallel. To measure through the cache token counts before choosing.
• Run store and retention. Where run records live, how long they stay, and how the recent-runs page reads them.
• Oversized diffs. What happens when the diff doesn't fit the model's context. Lockfiles already collapse to one line each, as in hive-forge diff.
• Budget. Any per-run or per-day spend limit on top of the concurrency cap.
• Diff-leaves-swarm policy. On the API key and ACP runtimes the diff goes to a third-party provider. Which repos may use which runtime, and whether anything gets redacted.
• Prompt injection. A PR's diff and commit messages are model input, so a PR can try to talk the classifier into approving it.
• The "no tools" mechanism on ACP. How the ACP runtime runs a session with no tools at all. The claude runtimes pass --tools "".
• Credential paths. Where in bao the ACP and subscription runtimes' credentials live. The API key is at secret/swarm/services/review-classifier/anthropic-api-key, field api_key, and the service account's forge token at secret/swarm/services/review-classifier/forge/token.
• Model naming across runtimes. How the swarm-level model setting handles runtime-specific model names.
• ACP cost labels. How an ACP run's cost gets a repo and PR label, so it reads from the metrics store like the claude runtimes' cost.
• Reviewer handoff. How a later reviewer, human or agent, learns the classifier passed a PR before it starts its own review.
• Try-out branch access. The service account's access for pushing try-out branches, beyond what a repo admin gives it for reviews.
• Commit author for try-out PRs. How the classifier derives the commit author's name and email from the swarm UI login.