The try-out page gets an "open a PR" button. It posts the page's classifications to the new POST /api/review-classifier/open-pr, which merges them by name into the repo's rules file on the default branch and has the review-classifier forge account commit the result on a new branch, review-classifier/<user>-<8 hex>, through the contents API. The commit's author is the forge account named like the session's Remote-User, read with the controller's admin token. The answer links to the forge's compare page, where the user opens the PR as themselves. The push uses the account's existing token from the store (swarm/services/review-classifier/forge/token, scope write:repository), so the account needs write access to the repo; a refused push answers 403 naming that. In a YAML rules file, the entries the merge leaves alone keep their text, as long as the result reads back as the merged list; otherwise the file is written out whole. DefaultBranchFiles now carries each file's blob sha, which the forge needs to replace a file, and the repo's web URL for the link. Fixes #5259 Refs #5096
616 lines
30 KiB
Markdown
616 lines
30 KiB
Markdown
# Review classifier
|
|
|
|
> ⚠️ **PLANNED, NOT IMPLEMENTED YET.** This page describes the full target
|
|
> shape of the review classifier at component level, and none of it exists
|
|
> in the tree today. The forge epic "pr review classifier pipeline" tracks
|
|
> the work. Details land here as each part ships, and this banner goes away
|
|
> when the epic closes.
|
|
|
|
The review classifier checks a pull request against short, named yes/no
|
|
rules that the repo itself defines, and posts the result as a forge review.
|
|
A small model answers one rule at a time, so mundane rule checks come out
|
|
the same way on every PR.
|
|
|
|
It doesn't replace a full review by a human or agent reviewer. A swarm can
|
|
request that review only once the classifier has passed the PR, and use a
|
|
larger model there. The classifier takes the cheap, repetitive checks off
|
|
that reviewer.
|
|
|
|
## Repo configuration
|
|
|
|
A repo opts in with a rules file at its root, in YAML as
|
|
`.hyperhive/review-classifier.yml` or in JSON as
|
|
`.hyperhive/review-classifier.json`. It holds one of the two; with both, the
|
|
rules don't load. `swarm-controller` always reads the file from the repo's
|
|
**default branch**, never from the PR, so a PR can't edit the rules it's
|
|
judged by. It reads it with its own forge token on every load, with no cache,
|
|
so a rule merged to the default branch counts from the next load on.
|
|
|
|
The file holds one key, `classifications`: a list of one to ten entries, each
|
|
with exactly these four keys:
|
|
|
|
- `name`: short and unique in the repo, shown in the result table
|
|
- `description`: the yes/no question the model answers about the PR
|
|
- `threshold`: the certainty, 0 to 100, a verdict needs to count
|
|
- `polarity`: `positive` means a match is the wanted outcome, and
|
|
`negative` means a match is a violation
|
|
|
|
Any other key, at the top or in an entry, is an error: a misspelt key fails
|
|
the load instead of vanishing. An empty list, more than ten entries, an empty
|
|
name or description, and two entries with one name are errors too. Nothing
|
|
falls back to a pass: a repo whose file doesn't load has no rules.
|
|
|
|
An example:
|
|
|
|
```yaml
|
|
classifications:
|
|
- name: datetime
|
|
description: All datetimes carry time zone information.
|
|
polarity: positive
|
|
threshold: 80
|
|
- name: no-ads
|
|
description: >-
|
|
A commit message contains a co-authored-by line or other
|
|
advertising for an AI model.
|
|
polarity: negative
|
|
threshold: 90
|
|
- name: bao-first
|
|
description: Any added secret reaches its reader through the bao store.
|
|
polarity: positive
|
|
threshold: 70
|
|
- name: docs-updated
|
|
description: >-
|
|
The PR updates the operator-facing docs where the code change
|
|
needs it.
|
|
polarity: positive
|
|
threshold: 60
|
|
```
|
|
|
|
## Trigger and lifecycle
|
|
|
|
The classifier has a dedicated forge service account, `review-classifier`
|
|
(see [The service account](#the-service-account)). Requesting that account
|
|
as a reviewer on a PR starts a run:
|
|
|
|
1. The forge sends the `pull_request_review_request` delivery to
|
|
[`swarm-controller`](../../swarm-controller/README.md), the swarm's one
|
|
webhook owner, on its instance-wide hook `/webhook/forge/review-request`.
|
|
A delivery for any other reviewer, or a removed request, does nothing.
|
|
2. The controller reads the PR and the repo's rules from the default branch,
|
|
and hands the run to the classifier service as one queue request on
|
|
`$SWARM.review-classifier.try-out`, the subject a try-out uses.
|
|
3. The classifier runs every classification against the PR's commits and
|
|
diff.
|
|
4. The controller posts the result on the PR as the service account's
|
|
review, on the commit it read.
|
|
|
|
A verdict belongs to the commit it ran on. A push starts no run. Once the
|
|
PR's head moves, requesting the service account as a reviewer again starts a
|
|
fresh run on the new head.
|
|
|
|
The re-request also retires the account's earlier review. The forge clears
|
|
the `official` flag on a reviewer's earlier reviews when it's requested
|
|
again, and only official reviews count toward the required approvals or
|
|
block a merge. Removing the request before the new review lands restores
|
|
the earlier one. A repo whose branch protection also dismisses stale approvals (see
|
|
[PR review gate](../process/pr-review-gate.md)) stops counting an approval
|
|
on an older commit even without a re-request.
|
|
|
|
How many requested reviews run at once is a swarm-wide setting, two by
|
|
default (see [Swarm-level settings](#swarm-level-settings)). Up to eight more
|
|
wait in the controller's memory, a bound that isn't configurable, so a
|
|
controller restart drops them, and nothing gets posted for them. Waiting
|
|
reviews start in no guaranteed order. A request for a PR and head that's
|
|
already running or waiting starts nothing. A request with every running place
|
|
and all eight waiting places taken starts no run: the account posts a
|
|
`**No verdict:**` comment saying so.
|
|
|
|
## Components
|
|
|
|
### The classifier service
|
|
|
|
A swarm-level service in its own container, on whichever host the operator
|
|
puts it. It talks to `swarm-controller` through async messages only, and
|
|
gets its secrets from the swarm's secret store, bao
|
|
([`bao.md`](bao.md), [`credentials.md`](credentials.md)).
|
|
|
|
It looks like an agent from the inside, with three things missing: it has no
|
|
full agent identity, no state directory and no tools. Every model session is
|
|
ephemeral, and nothing carries over from one run to the next.
|
|
|
|
On the API key runtime, each classification is one `claude --print` the
|
|
service spawns through the `hive-claude` crate directly, with no session to
|
|
resume, no tools (`--tools ""`), no MCP server (`--strict-mcp-config` and no
|
|
config) and an empty directory of its own. The prompt reaches it on stdin as
|
|
one user message in `--input-format stream-json`, so that it can carry content
|
|
blocks (see [Shared context](#shared-context)).
|
|
|
|
On the ACP runtime, each classification is one prompt on a new `opencode acp`
|
|
session. Its config and opencode's state live in a directory of its own,
|
|
removed after it. Its working directory is one empty directory that every
|
|
classification shares: opencode writes that path into the system message, so
|
|
one path keeps the system message the same from one classification to the
|
|
next, and the provider's prompt cache can hold. The service drives it through
|
|
[`hive-runtime`](../../hive-runtime/README.md)'s `AcpRuntime`, as the subagent
|
|
daemon does a one-run subagent, and stops the agent when the classification
|
|
ends. The agent has no tool. opencode runs some tools, `read` among them,
|
|
without asking the client, so the service sets opencode's own permission to
|
|
deny every tool (`OPENCODE_PERMISSION={"*":"deny"}`), which leaves the model
|
|
with no tool offered at all, and refuses any permission request on top. The
|
|
config it writes for opencode names one provider and model, from the settings,
|
|
and turns off opencode's title call, which would send the prompt a second
|
|
time. It also defines the agent the session runs as, whose prompt is the
|
|
classifier's system prompt, so opencode sends it as the system message, with
|
|
only its environment block (model, directory, platform, date) after it. The
|
|
first user message holds the PR alone. That agent has no `permission` of its
|
|
own: opencode merges an agent's permission over `OPENCODE_PERMISSION`, so one
|
|
there would offer the model tools again.
|
|
|
|
The service is `swarm-review-classifier`, and its container is the one
|
|
[`services.md`](services.md#review-classifier) describes. It holds no forge
|
|
credential: `swarm-controller` reads the PR and sends it along.
|
|
|
|
### Swarm-level settings
|
|
|
|
The runtime, the model, the effort and how many requested reviews run at once
|
|
are one document for the whole swarm. `swarm-controller` keeps it in its
|
|
`review-classifier-settings` KV bucket under the key `all`, and no other
|
|
service reads it. With no document stored, the classifier runs on the API key
|
|
runtime on `haiku` at claude's default effort, two reviews at a time. The
|
|
controller ignores a stored field it doesn't know, and a document it still
|
|
can't decode counts as none, with a warning in its log, until the next PUT
|
|
replaces it.
|
|
|
|
`GET /api/review-classifier/settings` answers the settings in force:
|
|
|
|
```json
|
|
{
|
|
"model": "haiku",
|
|
"effort": null,
|
|
"max_concurrent_reviews": 2,
|
|
"runtime": "api-key",
|
|
"acp": null
|
|
}
|
|
```
|
|
|
|
- `model`: the model every classification on the API key runtime runs on, as
|
|
claude's `--model` takes it: one word with no whitespace, not starting with
|
|
`-`
|
|
- `effort`: `low`, `medium`, `high`, `xhigh` or `max`; `null` runs at the
|
|
runtime's own default. On the ACP runtime it applies only where the agent
|
|
offers it
|
|
- `max_concurrent_reviews`: 1 to 8, how many requested reviews run at once.
|
|
Try-outs don't count toward it. Up to eight more reviews wait for a place
|
|
- `runtime`: `api-key` or `acp` (see [Runtimes](#runtimes))
|
|
- `acp`: the provider the ACP runtime runs on, or `null`. It stays stored
|
|
while the runtime is `api-key`, so switching back and forth needs no
|
|
re-entry. It holds:
|
|
- `base_url`: the provider's OpenAI-compatible API root, `https://` only
|
|
- `model`: the model as the provider names it, with no whitespace
|
|
- `prices`: what the provider charges in USD per million tokens,
|
|
`input` and `output`, and optionally `cache_read` and `cache_write`, which
|
|
otherwise count as `input`. A price is 0 or more
|
|
|
|
`PUT /api/review-classifier/settings` takes any of these fields, changes only
|
|
those, and answers the whole document it stored. A field left out keeps its
|
|
stored value; `null` sets `effort` or `acp` to none. A form that sends the
|
|
model, effort and limit alone therefore leaves the runtime as it was. No secret
|
|
goes in the document: an `api_key` field is an unknown field.
|
|
|
|
The controller reads the settings for every run and sends the runtime, model
|
|
and effort in the run request, so a change applies from the next run with no
|
|
restart. A try-out reads them when it's posted, a review when it gets its
|
|
place. Lowering the limit lets the runs already going finish. A run request
|
|
with no model runs on `haiku`, and one with no runtime on the API key.
|
|
|
|
The PUT needs the `Remote-User` header the gateway sets, as the try-out POST
|
|
does. Every error is problem+json with the reason in `detail`:
|
|
|
|
- **400** for a body that isn't JSON or isn't this shape: an unknown field, a
|
|
`null` other than for `effort` or `acp`, an effort outside the five levels,
|
|
a `max_concurrent_reviews` that isn't a whole number, a runtime other than
|
|
the two, or an `acp` without `base_url`, `model` or `prices`. Also **400**
|
|
when the document the PUT would store breaks a rule: an empty `model`, one
|
|
with whitespace or one starting with `-`, a `max_concurrent_reviews` outside
|
|
1 to 8, an `acp.base_url` that isn't `https://`, an empty `acp.model` or one
|
|
with whitespace, a negative price, or the `acp` runtime with no `acp`
|
|
- **401** for a PUT with no `Remote-User`
|
|
- **503** when this host has no swarm queue, or it isn't connected
|
|
- **500** when the controller can't read or write the settings
|
|
|
|
When the controller can't read the settings, a try-out answers the same 503
|
|
or 500, and a review posts a no-verdict comment. The operator writes each
|
|
runtime's key by hand in the secret store
|
|
([`services.md`](services.md#review-classifier)).
|
|
|
|
The per-agent equivalent is [`inference.md`](inference.md), where the
|
|
controller writes model and effort to a queue KV bucket
|
|
(`swarm-queue-client/src/agent_inference.rs`) and each agent reads its own
|
|
key.
|
|
|
|
### The service account
|
|
|
|
A dedicated forge user, separate from swarm-controller's site-admin account
|
|
and from every agent's account. It's the reviewer someone requests, it
|
|
authors every classifier review, and it pushes the branches the try-out page
|
|
creates.
|
|
|
|
`swarm-controller` creates it as `review-classifier`, with the email
|
|
`review-classifier@services.hyperhive.local` and no repo creation, and mints
|
|
its one token, `swarm-review-classifier` with scope `write:repository`, into
|
|
the store at `swarm/services/review-classifier/forge/token`. A pass at start
|
|
and every five minutes re-mints when the token is missing or no longer
|
|
matches the forge. The controller refuses an existing account by that name
|
|
with any other email. The controller posts the reviews and pushes the try-out
|
|
branches with that token, and the classifier service never sees it.
|
|
|
|
**Giving a repo access is a repo admin's step, by hand.** Nothing grants
|
|
it automatically. In the repo's settings, add `review-classifier` as a
|
|
collaborator:
|
|
|
|
- with **read**, so someone can request it as a reviewer, and its reviews show,
|
|
but its approval doesn't count
|
|
- with **write**, so its approval counts toward the required approvals and
|
|
its request-changes blocks the merge where the branch protection blocks on
|
|
rejected reviews. Where the branch protection restricts approvals to an
|
|
allowlist, the account counts only once it's on that list, and read access
|
|
is then enough for reviews.
|
|
|
|
The try-out page's **open a PR** pushes a branch to the repo itself, so it
|
|
needs **write** whatever the branch protection says.
|
|
|
|
The repo also needs a rules file on its default branch (see
|
|
[Repo configuration](#repo-configuration)). Without one, a request posts a
|
|
comment saying so.
|
|
|
|
## The classification run
|
|
|
|
### Shared context
|
|
|
|
Every classification in a run sees the same context, in the same order:
|
|
|
|
1. a short system prompt
|
|
2. the PR's commits, each with its full message
|
|
3. the full diff
|
|
|
|
Only the last part differs per classification: that classification's
|
|
description and the task to classify the PR against it. The identical prefix
|
|
lets the runtime's prompt cache reuse it across questions.
|
|
|
|
On the API key runtime the commits and diff are one content block of the user
|
|
message and the question another, with a cache breakpoint at the end of the
|
|
first. The cache writes an entry only where a breakpoint is, and `claude` sets
|
|
one only after the last block, so that breakpoint is what lets the next
|
|
question read the commits and diff from the cache. The PR stays in the user
|
|
message rather than the system prompt, which tells the model to judge it as
|
|
data. A shared context shorter than the model's minimum cacheable length is
|
|
not cached at all.
|
|
|
|
No object hash is in that prefix: the header names the repo and the PR number,
|
|
not the head commit, each commit's heading gives its position instead of its
|
|
sha, and the diff loses each file's `index <hash>..<hash>` line, keeping its
|
|
`diff --git` paths and every hunk. A rebase rewrites all of those hashes, so
|
|
with them in, a rebased PR with the same messages and hunks would miss the
|
|
cache from the first line. The head commit still goes to the controller,
|
|
which posts the review against it.
|
|
|
|
### One question, one answer
|
|
|
|
The model answers each classification in JSON. The answer holds:
|
|
|
|
- a **verdict**, one of _match_, _no match_ or _not applicable to this PR_
|
|
- a **certainty** from 0 to 100, how sure the model is of that verdict
|
|
- a **reason**; the classifier truncates one over 500 characters to 499 plus
|
|
`…`, and keeps the verdict and certainty
|
|
|
|
An answer:
|
|
|
|
```json
|
|
{
|
|
"verdict": "no-match",
|
|
"certainty": 92,
|
|
"reason": "Two new timestamps use naive datetimes without a zone."
|
|
}
|
|
```
|
|
|
|
The verdict field holds `match`, `no-match` or `not-applicable`, and the
|
|
certainty field a whole number. A runtime error, a timeout, or an answer that
|
|
isn't that JSON object leaves the classification without a verdict: it's listed
|
|
with its error, and doesn't count. A run with no key for its runtime, or in which no
|
|
classification got an answer, fails as a whole, and shows its error instead of a
|
|
verdict.
|
|
|
|
### Aggregation into the verdict
|
|
|
|
The classifier collects the answers into one result, keyed by
|
|
classification name, and logs it. A classification counts toward the
|
|
overall verdict only if it's applicable and its certainty reaches the
|
|
threshold. The table still lists every classification below its threshold or
|
|
not applicable, marked as excluded.
|
|
|
|
A counted classification fails when its verdict lands on the wrong side of
|
|
its polarity: _no match_ on a positive one, _match_ on a negative one. Any
|
|
failing classification makes the verdict **request changes**. Otherwise the
|
|
verdict is **approve**, including when the run counted no classification at
|
|
all.
|
|
|
|
### The review
|
|
|
|
The service account posts the verdict as a forge review, approve or request
|
|
changes, and the table is the review's body. An illustrative table:
|
|
|
|
| classification | polarity | verdict | certainty | threshold | counted | reason |
|
|
| -------------- | -------- | -------------- | --------- | --------- | ------- | -------------------------------------------- |
|
|
| datetime | positive | no match | 92 | 80 | yes ❌ | Two new timestamps use naive datetimes. |
|
|
| no-ads | negative | no match | 99 | 90 | yes ✅ | No commit message carries an AI attribution. |
|
|
| bao-first | positive | not applicable | 95 | 70 | no | The PR adds no secret. |
|
|
| docs-updated | positive | match | 41 | 60 | no | Below threshold. |
|
|
|
|
**Overall: request changes.**
|
|
|
|
The approval is a real one: it counts toward a repo's required approvals
|
|
like any other ([`pr-review-gate.md`](../process/pr-review-gate.md)). The
|
|
Repos that rely on it typically raise their required approval count to
|
|
account for it.
|
|
|
|
A run with no clean verdict never approves. When the PR or the rules don't
|
|
load, the run fails, the classifier answers a different number of
|
|
classifications than the controller sent it, or **any** classification has no answer,
|
|
the account posts a **comment** instead, starting with `**No verdict:**` and the reason,
|
|
followed by the table when there is one. A comment neither approves nor
|
|
blocks the merge: nine timeouts and one pass post a comment, not an
|
|
approval.
|
|
|
|
The controller flattens every cell to one line, with `|`, backticks, brackets, `<`, `>`,
|
|
`&`, `#` and other markup characters escaped, and a zero-width space after
|
|
each `@`, `#`, `!`, `://` and `www.`, so a model's `reason` can't break the
|
|
table, embed markup, mention anyone, reference an issue or PR, or carry a
|
|
link.
|
|
|
|
## Runtimes
|
|
|
|
The classifier runs on one of two runtimes, the settings' `runtime`:
|
|
|
|
- **API key** (`api-key`, the default): the `claude` CLI on the Anthropic API
|
|
key at `secret/swarm/services/review-classifier/anthropic-api-key`, at
|
|
`model` and `effort`
|
|
- **ACP** (`acp`): `opencode` on the provider `acp` names, with the provider
|
|
key at `secret/swarm/services/review-classifier/acp-provider`, at
|
|
`acp.model`
|
|
|
|
Model names don't carry across runtimes. A claude alias means nothing to an
|
|
ACP provider, so each runtime has a model field of its own.
|
|
|
|
To switch to ACP, an admin:
|
|
|
|
1. writes the provider key in the secret store's UI under the OIDC login, at
|
|
`secret/swarm/services/review-classifier/acp-provider`, field `api_key`
|
|
2. PUTs the runtime and the provider:
|
|
|
|
```json
|
|
{
|
|
"runtime": "acp",
|
|
"acp": {
|
|
"base_url": "https://llm.example/v1",
|
|
"model": "Qwen/Qwen3-35B",
|
|
"prices": { "input": 0.5, "output": 2 }
|
|
}
|
|
}
|
|
```
|
|
|
|
The next run goes to that provider. `{"runtime": "api-key"}` switches back and
|
|
keeps the provider for next time.
|
|
|
|
A split deploy runs on the API key: a classifier older than the ACP runtime
|
|
ignores `runtime` and runs every classification on `claude`, and a controller
|
|
older than it ignores the stored `runtime` and `acp` and sends neither.
|
|
|
|
A third runtime, **claude subscription** (the `claude` CLI on a Claude
|
|
subscription login), comes later.
|
|
|
|
> **Open point.** The subscription runtime needs a login, which is the one
|
|
> piece of durable state the service would keep. A swarm-level claude
|
|
> subscription login may have to exist before this runtime works.
|
|
|
|
## Swarm UI
|
|
|
|
Two pages in the [swarm UI](ui.md), `admins` only like the rest of it.
|
|
|
|
### Recent runs
|
|
|
|
One card per run: the repo, the PR, and how many classifications ran.
|
|
Selecting a card expands it to show each classification's result, the same
|
|
rows as the review table.
|
|
|
|
### Try-out
|
|
|
|
A page for writing classification prompts without touching the repo. Pick a
|
|
repo and a PR, write or edit a classification, and run it against that PR;
|
|
change the prompt and run it again until it answers the way you want.
|
|
Nothing reaches the forge while you iterate.
|
|
|
|
The page posts the run to `swarm-controller` (`POST
|
|
/api/review-classifier/try-out`), which reads the PR with its own forge token
|
|
and sends it to the classifier as one queue request on
|
|
`$SWARM.review-classifier.try-out`. The page then reads the result back (`GET
|
|
/api/review-classifier/try-out/{id}`): a run can take longer than a request
|
|
stays open. The controller keeps the result in memory for an hour, and the
|
|
finished run goes into the [run store](#run-store-and-retention).
|
|
|
|
`GET /api/review-classifier/rules?repo=owner/name` returns the repo's own
|
|
rules, for the page to load into its editor; a run posts them like typed
|
|
ones. The route answers:
|
|
|
|
- **200** with `path` (the file read), `branch` (the default branch),
|
|
`commit` (the commit the route read the file at) and `classifications`, in
|
|
the file's order and in the POST's shape
|
|
- **400** for a `repo` that's missing or not `owner/name`
|
|
- **404** for a repo the forge doesn't know; a default branch with no commit,
|
|
in an empty repo or when the default branch doesn't exist; or a default
|
|
branch with neither file
|
|
- **422** for both files on the default branch, or a file that doesn't parse
|
|
or breaks a rule from [Repo configuration](#repo-configuration); the
|
|
`detail` names the file and what's wrong
|
|
- **503** when this host has no forge token yet, or the forge failed or
|
|
didn't answer within two minutes
|
|
|
|
The page's repo and PR fields offer values to pick from, and stay free text,
|
|
since a run also works on a merged or closed PR. Two routes feed them:
|
|
|
|
- `GET /api/pulls/repos` lists the repos with at least one open PR, as
|
|
`owner/name`.
|
|
- `GET /api/pulls?repo=owner/name` lists the repo's open PRs, highest number
|
|
first, each with `number`, `title` and `head_sha`. It answers **400** for a
|
|
`repo` that's missing or not `owner/name`, **404** for a repo the forge
|
|
doesn't know, and **503** when this host has no forge token yet, or the
|
|
forge failed or didn't answer within two minutes.
|
|
|
|
Every error is problem+json with the reason in `detail`.
|
|
|
|
When it's ready, **open a PR** (`POST /api/review-classifier/open-pr`, with
|
|
the repo and the classifications) has the service account push a branch that
|
|
adds the classifications to the repo's rules, with you, the swarm UI user, as
|
|
commit author. The page then links you to the forge's compare page for that
|
|
branch, where you log in as usual and open the PR yourself.
|
|
|
|
- **The branch** is `review-classifier/<you>-<8 hex digits>`, made from the
|
|
default branch's head with one commit, through the forge's contents API.
|
|
- **The file** is the rules file on the default branch, merged by name: a
|
|
classification named like an entry replaces it, a new name goes at the end,
|
|
and every other entry stays. A repo with no rules file gets
|
|
`.hyperhive/review-classifier.yml`. In a YAML file, the entries the merge
|
|
leaves alone keep their text, comments included. The merged file has to
|
|
pass the same checks the `rules` route makes.
|
|
- **The author** is the forge account named like your swarm UI login. A forge
|
|
account made by an SSO login carries the SSO username, so it's yours once
|
|
you've logged in to the forge. The commit carries its full name, or the
|
|
login when it has none, and its email. The committer is the service
|
|
account.
|
|
- **The token** is the service account's own, read from the store for each
|
|
push (see [The service account](#the-service-account)), so the account
|
|
needs **write** access to the repo.
|
|
|
|
The route answers **201** with `branch`, `path` (the file written) and
|
|
`compare_url`, or:
|
|
|
|
- **400** for a body the try-out POST would refuse in its repo or
|
|
classifications
|
|
- **401** without the `Remote-User` header the gateway sets
|
|
- **403** when the forge refuses the push: the service account isn't a
|
|
collaborator with write access
|
|
- **404** for a repo the forge doesn't know, or one with no commit on its
|
|
default branch
|
|
- **409** when the forge has no account named like you yet, or the rules
|
|
file changed on the default branch during the push
|
|
- **422** for a default branch holding both rules files or one the `rules`
|
|
route refuses; classifications the file already holds unchanged; or a
|
|
merge that breaks a rule, such as more than ten classifications
|
|
- **503** when this host has no forge token, the service account's token
|
|
isn't minted yet, or the forge failed or didn't answer within two minutes
|
|
|
|
## Run store and retention
|
|
|
|
`swarm-controller` records every run when it ends: a try-out when the
|
|
classifier's reply arrives, a requested review once the controller has posted
|
|
its review or the post has failed, and a request turned away because the
|
|
queue was full once its comment is on the forge or the post has failed. A review request for a head that's already
|
|
running or waiting starts nothing and isn't recorded. A run still going isn't
|
|
in the list, and a controller restart drops a run in progress.
|
|
|
|
The newest 100 runs are one JSON array under the key `recent` in the
|
|
`review-classifier-runs` KV bucket on the swarm queue, which only the
|
|
controller reads or writes. The controller creates the bucket on first use and
|
|
adds each run with a compare-and-set write, so the list survives a controller
|
|
restart or a move to another host. Recording the 101st run drops the oldest;
|
|
there's no time-based retention. The controller leaves a run it can't decode, such
|
|
as one a newer build recorded, out of the list with a warning in its log. It
|
|
keeps the run in the store as it was and counts it toward the 100. A document
|
|
that isn't a list reads as no runs. The controller logs a failed write, and it
|
|
changes nothing on the forge.
|
|
|
|
`GET /api/review-classifier/runs` returns the list, newest first. Like the
|
|
try-out GET, the gateway's `admins` rule gates it, and the controller checks no
|
|
header. Each run has:
|
|
|
|
- `id`: a try-out's is the id its POST answered with
|
|
- `trigger`: `try-out` or `review`
|
|
- `repo` (`owner/name`), `pr`, and `requester`: the swarm UI user who started
|
|
the try-out, or the forge user who requested the review
|
|
- `head_sha`: the commit the run read, absent when the PR wasn't read, as for
|
|
an unreadable PR or a full queue
|
|
- `started_unix` and `finished_unix`: a try-out starts at its POST, a review
|
|
once it has a run place
|
|
- `outcome`: `approved`, `changes-requested`, `no-verdict`, `queue-full` or
|
|
`post-failed` for a review; `done` or `failed` for a try-out
|
|
- `detail`: the reason, for every outcome but `approved`,
|
|
`changes-requested` and `done`. For `post-failed` it's the run's own reason,
|
|
when it has one, then why the post failed.
|
|
- `rows` and `overall`, as the try-out GET returns them, when the classifier
|
|
answered
|
|
- `review_url`: the posted review, when the forge names its URL
|
|
|
|
A run holds no diff, no commit text and no token count or cost. The route
|
|
answers 503 when the controller has no swarm queue or can't reach the bucket,
|
|
and 500 when it can't read the list, both as problem+json.
|
|
|
|
Each recorded run is also one `info` event from `swarm-controller` with the
|
|
message `review classifier run finished`. The event carries the same fields,
|
|
with `rows` as JSON. It reaches VictoriaLogs through the host journal, where
|
|
each field arrives as `F_` and the name in capitals (`F_OUTCOME`, `F_REPO`), so
|
|
a run whose write failed is still there:
|
|
|
|
```console
|
|
$ swarm-logs query '_time:7d _SYSTEMD_UNIT:="swarm-controller.service" _msg:="review classifier run finished" | fields _time, F_TRIGGER, F_REPO, F_PR, F_OUTCOME, F_DETAIL, F_REVIEW_URL' --format json
|
|
```
|
|
|
|
Add a filter such as `F_OUTCOME:="post-failed"` before the `|` to narrow it.
|
|
|
|
## Observability and cost
|
|
|
|
The swarm UI shows no token counts, cache counts or cost, on any page. On the
|
|
claude runtimes, every classifier run starts its `claude` process with
|
|
`OTEL_RESOURCE_ATTRIBUTES` naming the repo and the PR, so the cost reads from
|
|
the metrics store by repo and PR. The process exports to the hive-tier collector on
|
|
the classifier's host, where that host runs one. On a subscription, cost is notional.
|
|
|
|
On the ACP runtime, the service computes each classification's cost itself,
|
|
from the settings' `acp.prices` and the token counts the agent reports for the
|
|
prompt, and adds it to the counter `hyperhive.review_classifier.cost.usage`,
|
|
labelled `repo`, `pr`, `model` (the provider's) and `currency` (`USD`). It
|
|
exports to the same collector `claude` does. A classification stopped at its
|
|
deadline, or whose agent reports no token counts, adds nothing. opencode
|
|
reports no cost of its own for a provider without prices in its config, which
|
|
is why the prices are a setting.
|
|
|
|
Each such classification also adds the tokens it used to the counter
|
|
`hyperhive.review_classifier.token.usage`, with the same labels bar `currency`
|
|
and a `type` of `input`, `output`, `cacheRead` or `cacheCreation`. Those are
|
|
the types of `claude_code.token.usage`, which `claude` exports on the claude
|
|
runtimes, so the cache hit rate reads the same way on every runtime: `cacheRead`
|
|
over the sum of `input`, `cacheRead` and `cacheCreation`.
|
|
|
|
The classifier logs each run's result to its journal, which ships to
|
|
VictoriaLogs like every swarm service container's
|
|
([`services.md`](services.md#logs-victorialogs)).
|
|
|
|
## Still open
|
|
|
|
These get settled, and filled in here, as the work reaches them:
|
|
|
|
- **Cache strategy.** Fully independent sessions per classification, or one
|
|
warm session and then the rest in parallel. To measure through the cache
|
|
token counts before choosing.
|
|
- **Oversized diffs.** What happens when the diff doesn't fit the model's
|
|
context. Lockfiles already collapse to one line each, as in `hive-forge diff`.
|
|
- **Budget.** Any per-run or per-day spend limit on top of the concurrency
|
|
cap.
|
|
- **Diff-leaves-swarm policy.** On the API key and ACP runtimes the diff
|
|
goes to a third-party provider. Which repos may use which runtime, and
|
|
whether anything gets redacted.
|
|
- **Prompt injection.** A PR's diff and commit messages are model input, so
|
|
a PR can try to talk the classifier into approving it.
|
|
- **Subscription credential.** Where in bao the subscription runtime's login
|
|
lives. The service account's forge token is at
|
|
`secret/swarm/services/review-classifier/forge/token`.
|
|
- **Reviewer handoff.** How a later reviewer, human or agent, learns the
|
|
classifier passed a PR before it starts its own review.
|