hyperhive/docs/swarm/review-classifier.md

Review classifier

⚠️ PLANNED, NOT IMPLEMENTED YET. This page describes the full target shape of the review classifier at component level, and none of it exists in the tree today. The forge epic "pr review classifier pipeline" tracks the work. Details land here as each part ships, and this banner goes away when the epic closes.

The review classifier checks a pull request against short, named yes/no rules that the repo itself defines, and posts the result as a forge review. A small model answers one rule at a time, so mundane rule checks come out the same way on every PR.

It doesn't replace a full review by a human or agent reviewer. A swarm can request that review only once the classifier has passed the PR, and use a larger model there. The classifier takes the cheap, repetitive checks off that reviewer.

Repo configuration

A repo opts in with a config file in a .hyperhive/ directory at its root, in YAML or JSON. The classifier always reads it from the repo's default branch, never from the PR, so a PR can't edit the rules it's judged by.

Each classification has:

• a name, short and unique in the repo, shown in the result table
• a description: the yes/no question the model answers about the PR
• a certainty threshold from 0 to 100
• a polarity: positive means a match is the wanted outcome, and negative means a match is a violation

An illustrative example. The file name and every field name in it are placeholders, not a fixed schema:

# illustrative only: file name and field names aren't fixed
classifications:
  - name: datetime
    description: All datetimes carry time zone information.
    polarity: positive
    threshold: 80
  - name: no-ads
    description: >-
      A commit message contains a co-authored-by line or other
      advertising for an AI model.
    polarity: negative
    threshold: 90
  - name: bao-first
    description: Any added secret reaches its reader through the bao store.
    polarity: positive
    threshold: 70
  - name: docs-updated
    description: >-
      The PR updates the operator-facing docs where the code change
      needs it.
    polarity: positive
    threshold: 60

Trigger and lifecycle

The classifier has a dedicated forge service account. Requesting that account as a reviewer on a PR starts a run:

1. The forge sends the review-request webhook to swarm-controller, the swarm's one webhook owner.
2. The controller hands the run to the classifier service as an async message.
3. The classifier runs every classification in the repo's config, read from the default branch, against the PR's commits and diff.
4. The service account posts the result on the PR as a review.

A verdict belongs to the commit it ran on. Once the PR's head moves, requesting the service account as a reviewer again starts a fresh run on the new head.

That only holds if the bot's review on an older commit stops counting once the head moves. Either the repo enables Forgejo's stale-review dismissal (see PR review gate), or the service dismisses its own earlier review when it posts a new one. Where neither applies, an old approval keeps counting and an old request-changes keeps blocking.

Runs go in parallel, up to a maximum set at swarm level.

Components

The classifier service

A swarm-level service in its own container, on whichever host the operator puts it. It talks to swarm-controller through async messages only, and gets its secrets from the swarm's secret store, bao (bao.md, credentials.md).

It looks like an agent from the inside, with three things missing: it has no full agent identity, no state directory and no tools. Every model session is ephemeral, and nothing carries over from one run to the next.

The model sessions run through hive-runtime, the crate hive-agent and hive-subagent-mcp already drive their turns through: the Runtime trait in hive-runtime/src/lib.rs, with ClaudeRuntime and AcpRuntime behind it. That trait runs turns on a durable session, and the subagent daemon's claude path (hive-subagent-mcp/src/session.rs) spawns claude directly. A refactor first moves the cancellable per-turn claude session into hive-runtime and moves the subagent daemon onto it; the classifier builds on that shared seam.

Swarm-level settings

The operator sets the classifier's settings for the whole swarm:

• the runtime (see Runtimes) and its credentials
• the model and effort, and related inference settings
• the maximum number of concurrent runs

The per-agent equivalent is inference.md, where the controller writes model and effort to a queue KV bucket (swarm-queue-client/src/agent_inference.rs) and each agent reads its own key.

The service account

A dedicated forge user, separate from swarm-controller's site-admin account and from every agent's account. It's the reviewer someone requests, it authors every classifier review, and it pushes the branches the try-out page creates.

The classification run

Shared context

Every classification in a run sees the same context, in the same order:

1. a short system prompt
2. the PR's commits, each with its full message
3. the full diff

Only the last part differs per classification: that classification's description and the task to classify the PR against it. The identical prefix lets the runtime's prompt cache reuse it across questions.

One question, one answer

The model answers each classification in JSON. The answer holds:

• a verdict, one of match, no match or not applicable to this PR
• a certainty from 0 to 100, how sure the model is of that verdict
• a reason, limited in length

An illustrative answer, with placeholder field names:

{
  "verdict": "no-match",
  "certainty": 92,
  "reason": "Two new timestamps use naive datetimes without a zone."
}

Aggregation into the verdict

The classifier collects the answers into one result, keyed by classification name, and logs it. A classification counts toward the overall verdict only if it's applicable and its certainty reaches the threshold. The table still lists every classification below its threshold or not applicable, marked as excluded.

A counted classification fails when its verdict lands on the wrong side of its polarity: no match on a positive one, match on a negative one. Any failing classification makes the verdict request changes. Otherwise the verdict is approve, including when the run counted no classification at all.

The review

The service account posts the verdict as a forge review, approve or request changes, and the table is the review's body. An illustrative table:

classification	polarity	verdict	certainty	threshold	counted	reason
datetime	positive	no match	92	80	yes ❌	Two new timestamps use naive datetimes.
no-ads	negative	no match	99	90	yes ✅	No commit message carries an AI attribution.
bao-first	positive	not applicable	95	70	no	The PR adds no secret.
docs-updated	positive	match	41	60	no	Below threshold.

Overall: request changes.

The approval is a real one: it counts toward a repo's required approvals like any other (pr-review-gate.md). The Repos that rely on it typically raise their required approval count to account for it.

Runtimes

The classifier runs on one of three runtimes, chosen at swarm level:

• claude subscription: the claude CLI on a Claude subscription login
• API key: the claude CLI on an API key
• ACP: an ACP agent and its provider, as for an ACP agent today

Model names don't carry across runtimes. A claude alias means nothing to an ACP provider, so the model setting belongs to the runtime it names.

Open point. The subscription runtime needs a login, which is the one piece of durable state the service would keep. A swarm-level claude subscription login may have to exist before this runtime works.

Swarm UI

Two pages in the swarm UI, admins only like the rest of it.

Recent runs

One card per run: the repo, the PR, how many classifications ran, and the run's overall cost. Selecting a card expands it to show each classification's result, the same rows as the review table.

Try-out

A page for writing classification prompts without touching the repo. Pick a repo and a PR, write or edit a classification, and run it against that PR; change the prompt and run it again until it answers the way you want. Nothing reaches the forge while you iterate.

When it's ready, open a PR has the service account push a branch that adds the classification to the repo's config, with you, the swarm UI user, as commit author. The page then links you to the forge's new-PR page for that branch, where you log in as usual and open the PR yourself.

Observability and cost

Every run records its cost, which the recent-runs page shows. Where the cost comes from depends on the runtime:

• An ACP agent reports a running cost total (ReportedCost in hive-runtime/src/acp/stream.rs); a session's last total is its cost.
• The claude CLI reports token counts, including cache reads and writes. The dashboard already estimates a cost from token counts with a price table (modelPrices in dashboard.md).
• On a subscription, cost is notional.

The cache token counts also show whether the shared context gets the cache reuse it's meant to.

The classifier logs each run's result to its journal, which ships to VictoriaLogs like every swarm service container's (services.md).

Still open

These get settled, and filled in here, as the work reaches them:

• Cache strategy. Fully independent sessions per classification, or one warm session and then the rest in parallel. To measure through the cache token counts before choosing.
• Run store and retention. Where run records live, how long they stay, and how the recent-runs page reads them.
• Failure handling. What a runtime error, malformed JSON, a timeout or a rate limit does to a classification and to the verdict.
• JSON validation and the reason cap. The exact answer schema, and the reason's length limit.
• Oversized diffs. What happens when the diff doesn't fit the model's context, and whether lockfiles collapse as in hive-forge diff (hive-forge/src/verbs/diff.rs).
• Stale reviews. Whether the repo's stale-review dismissal setting retires the bot's review on an older commit, or the service dismisses its own earlier review when it posts a new one.
• Budget. Any per-run or per-day spend limit on top of the concurrency cap.
• Diff-leaves-swarm policy. On the API key and ACP runtimes the diff goes to a third-party provider. Which repos may use which runtime, and whether anything gets redacted.
• Prompt injection. A PR's diff and commit messages are model input, so a PR can try to talk the classifier into approving it.
• The "no tools" mechanism. How each runtime runs a session with no tools at all.
• Credential paths. Where in bao the runtime credentials and the service account's forge token live, and which component holds the token.
• Model naming across runtimes. How the swarm-level model setting handles runtime-specific model names.
• Cost method per runtime and telemetry labels. The exact cost source per runtime, and the resource attributes that keep the classifier's spend separate from the agents' on the existing dashboards.
• Async message medium. Which queue subjects or streams carry the messages between swarm-controller and the classifier.
• Reviewer handoff. How a later reviewer, human or agent, learns the classifier passed a PR before it starts its own review.
• Opt-in details. Beyond the config file: the service account's access to a repo, so someone can request it as reviewer and it can push try-out branches.
• Commit author for try-out PRs. How the classifier derives the commit author's name and email from the swarm UI login.
• Refactor scope. How much of hive-agent's and hive-subagent-mcp's runtime code moves into hive-runtime before the classifier uses it.