The review body's table is now: result | check | certainty / threshold | reason. The result cell is a symbol for what the row did to the review, after polarity (✅ counted and passed, ❌ counted and failed, ❔ not counted or no answer), then the model's verdict. The overall line carries ✅ / ❌. Polarity, threshold and counted are folded into those cells. The prompt asks for a reason under 350 characters; the enforced cap stays 500, so an overshoot is truncated less often.
625 lines
30 KiB
Markdown
625 lines
30 KiB
Markdown
# Review classifier
|
|
|
|
> ⚠️ **PLANNED, NOT IMPLEMENTED YET.** This page describes the full target
|
|
> shape of the review classifier at component level, and none of it exists
|
|
> in the tree today. The forge epic "pr review classifier pipeline" tracks
|
|
> the work. Details land here as each part ships, and this banner goes away
|
|
> when the epic closes.
|
|
|
|
The review classifier checks a pull request against short, named yes/no
|
|
rules that the repo itself defines, and posts the result as a forge review.
|
|
A small model answers one rule at a time, so mundane rule checks come out
|
|
the same way on every PR.
|
|
|
|
It doesn't replace a full review by a human or agent reviewer. A swarm can
|
|
request that review only once the classifier has passed the PR, and use a
|
|
larger model there. The classifier takes the cheap, repetitive checks off
|
|
that reviewer.
|
|
|
|
## Repo configuration
|
|
|
|
A repo opts in with a rules file at its root, in YAML as
|
|
`.hyperhive/review-classifier.yml` or in JSON as
|
|
`.hyperhive/review-classifier.json`. It holds one of the two; with both, the
|
|
rules don't load. `swarm-controller` always reads the file from the repo's
|
|
**default branch**, never from the PR, so a PR can't edit the rules it's
|
|
judged by. It reads it with its own forge token on every load, with no cache,
|
|
so a rule merged to the default branch counts from the next load on.
|
|
|
|
The file holds one key, `classifications`: a list of one to ten entries, each
|
|
with exactly these four keys:
|
|
|
|
- `name`: short and unique in the repo, shown in the result table
|
|
- `description`: the yes/no question the model answers about the PR
|
|
- `threshold`: the certainty, 0 to 100, a verdict needs to count
|
|
- `polarity`: `positive` means a match is the wanted outcome, and
|
|
`negative` means a match is a violation
|
|
|
|
Any other key, at the top or in an entry, is an error: a misspelt key fails
|
|
the load instead of vanishing. An empty list, more than ten entries, an empty
|
|
name or description, and two entries with one name are errors too. Nothing
|
|
falls back to a pass: a repo whose file doesn't load has no rules.
|
|
|
|
An example:
|
|
|
|
```yaml
|
|
classifications:
|
|
- name: datetime
|
|
description: All datetimes carry time zone information.
|
|
polarity: positive
|
|
threshold: 80
|
|
- name: no-ads
|
|
description: >-
|
|
A commit message contains a co-authored-by line or other
|
|
advertising for an AI model.
|
|
polarity: negative
|
|
threshold: 90
|
|
- name: bao-first
|
|
description: Any added secret reaches its reader through the bao store.
|
|
polarity: positive
|
|
threshold: 70
|
|
- name: docs-updated
|
|
description: >-
|
|
The PR updates the operator-facing docs where the code change
|
|
needs it.
|
|
polarity: positive
|
|
threshold: 60
|
|
```
|
|
|
|
## Trigger and lifecycle
|
|
|
|
The classifier has a dedicated forge service account, `review-classifier`
|
|
(see [The service account](#the-service-account)). Requesting that account
|
|
as a reviewer on a PR starts a run:
|
|
|
|
1. The forge sends the `pull_request_review_request` delivery to
|
|
[`swarm-controller`](../../swarm-controller/README.md), the swarm's one
|
|
webhook owner, on its instance-wide hook `/webhook/forge/review-request`.
|
|
A delivery for any other reviewer, or a removed request, does nothing.
|
|
2. The controller reads the PR and the repo's rules from the default branch,
|
|
and hands the run to the classifier service as one queue request on
|
|
`$SWARM.review-classifier.try-out`, the subject a try-out uses.
|
|
3. The classifier runs every classification against the PR's commits and
|
|
diff.
|
|
4. The controller posts the result on the PR as the service account's
|
|
review, on the commit it read.
|
|
|
|
A verdict belongs to the commit it ran on. A push starts no run. Once the
|
|
PR's head moves, requesting the service account as a reviewer again starts a
|
|
fresh run on the new head.
|
|
|
|
The re-request also retires the account's earlier review. The forge clears
|
|
the `official` flag on a reviewer's earlier reviews when it's requested
|
|
again, and only official reviews count toward the required approvals or
|
|
block a merge. Removing the request before the new review lands restores
|
|
the earlier one. A repo whose branch protection also dismisses stale approvals (see
|
|
[PR review gate](../process/pr-review-gate.md)) stops counting an approval
|
|
on an older commit even without a re-request.
|
|
|
|
How many requested reviews run at once is a swarm-wide setting, two by
|
|
default (see [Swarm-level settings](#swarm-level-settings)). Up to eight more
|
|
wait in the controller's memory, a bound that isn't configurable, so a
|
|
controller restart drops them, and nothing gets posted for them. Waiting
|
|
reviews start in no guaranteed order. A request for a PR and head that's
|
|
already running or waiting starts nothing. A request with every running place
|
|
and all eight waiting places taken starts no run: the account posts a
|
|
`**No verdict:**` comment saying so.
|
|
|
|
## Components
|
|
|
|
### The classifier service
|
|
|
|
A swarm-level service in its own container, on whichever host the operator
|
|
puts it. It talks to `swarm-controller` through async messages only, and
|
|
gets its secrets from the swarm's secret store, bao
|
|
([`bao.md`](bao.md), [`credentials.md`](credentials.md)).
|
|
|
|
It looks like an agent from the inside, with three things missing: it has no
|
|
full agent identity, no state directory and no tools. Every model session is
|
|
ephemeral, and nothing carries over from one run to the next.
|
|
|
|
On the API key runtime, each classification is one `claude --print` the
|
|
service spawns through the `hive-claude` crate directly, with no session to
|
|
resume, no tools (`--tools ""`), no MCP server (`--strict-mcp-config` and no
|
|
config) and an empty directory of its own. The prompt reaches it on stdin as
|
|
one user message in `--input-format stream-json`, so that it can carry content
|
|
blocks (see [Shared context](#shared-context)).
|
|
|
|
On the ACP runtime, each classification is one prompt on a new `opencode acp`
|
|
session. Its config and opencode's state live in a directory of its own,
|
|
removed after it. Its working directory is one empty directory that every
|
|
classification shares: opencode writes that path into the system message, so
|
|
one path keeps the system message the same from one classification to the
|
|
next, and the provider's prompt cache can hold. The service drives it through
|
|
[`hive-runtime`](../../hive-runtime/README.md)'s `AcpRuntime`, as the subagent
|
|
daemon does a one-run subagent, and stops the agent when the classification
|
|
ends. The agent has no tool. opencode runs some tools, `read` among them,
|
|
without asking the client, so the service sets opencode's own permission to
|
|
deny every tool (`OPENCODE_PERMISSION={"*":"deny"}`), which leaves the model
|
|
with no tool offered at all, and refuses any permission request on top. The
|
|
config it writes for opencode names one provider and model, from the settings,
|
|
and turns off opencode's title call, which would send the prompt a second
|
|
time. It also defines the agent the session runs as, whose prompt is the
|
|
classifier's system prompt, so opencode sends it as the system message, with
|
|
only its environment block (model, directory, platform, date) after it. The
|
|
first user message holds the PR alone. That agent has no `permission` of its
|
|
own: opencode merges an agent's permission over `OPENCODE_PERMISSION`, so one
|
|
there would offer the model tools again.
|
|
|
|
The service is `swarm-review-classifier`, and its container is the one
|
|
[`services.md`](services.md#review-classifier) describes. It holds no forge
|
|
credential: `swarm-controller` reads the PR and sends it along.
|
|
|
|
### Swarm-level settings
|
|
|
|
The runtime, the model, the effort and how many requested reviews run at once
|
|
are one document for the whole swarm. `swarm-controller` keeps it in its
|
|
`review-classifier-settings` KV bucket under the key `all`, and no other
|
|
service reads it. With no document stored, the classifier runs on the API key
|
|
runtime on `haiku` at claude's default effort, two reviews at a time. The
|
|
controller ignores a stored field it doesn't know, and a document it still
|
|
can't decode counts as none, with a warning in its log, until the next PUT
|
|
replaces it.
|
|
|
|
`GET /api/review-classifier/settings` answers the settings in force:
|
|
|
|
```json
|
|
{
|
|
"model": "haiku",
|
|
"effort": null,
|
|
"max_concurrent_reviews": 2,
|
|
"runtime": "api-key",
|
|
"acp": null
|
|
}
|
|
```
|
|
|
|
- `model`: the model every classification on the API key runtime runs on, as
|
|
claude's `--model` takes it: one word with no whitespace, not starting with
|
|
`-`
|
|
- `effort`: `low`, `medium`, `high`, `xhigh` or `max`; `null` runs at the
|
|
runtime's own default. On the ACP runtime it applies only where the agent
|
|
offers it
|
|
- `max_concurrent_reviews`: 1 to 8, how many requested reviews run at once.
|
|
Try-outs don't count toward it. Up to eight more reviews wait for a place
|
|
- `runtime`: `api-key` or `acp` (see [Runtimes](#runtimes))
|
|
- `acp`: the provider the ACP runtime runs on, or `null`. It stays stored
|
|
while the runtime is `api-key`, so switching back and forth needs no
|
|
re-entry. It holds:
|
|
- `base_url`: the provider's OpenAI-compatible API root, `https://` only
|
|
- `model`: the model as the provider names it, with no whitespace
|
|
- `prices`: what the provider charges in USD per million tokens,
|
|
`input` and `output`, and optionally `cache_read` and `cache_write`, which
|
|
otherwise count as `input`. A price is 0 or more
|
|
|
|
`PUT /api/review-classifier/settings` takes any of these fields, changes only
|
|
those, and answers the whole document it stored. A field left out keeps its
|
|
stored value; `null` sets `effort` or `acp` to none. A form that sends the
|
|
model, effort and limit alone therefore leaves the runtime as it was. No secret
|
|
goes in the document: an `api_key` field is an unknown field.
|
|
|
|
The controller reads the settings for every run and sends the runtime, model
|
|
and effort in the run request, so a change applies from the next run with no
|
|
restart. A try-out reads them when it's posted, a review when it gets its
|
|
place. Lowering the limit lets the runs already going finish. A run request
|
|
with no model runs on `haiku`, and one with no runtime on the API key.
|
|
|
|
The PUT needs the `Remote-User` header the gateway sets, as the try-out POST
|
|
does. Every error is problem+json with the reason in `detail`:
|
|
|
|
- **400** for a body that isn't JSON or isn't this shape: an unknown field, a
|
|
`null` other than for `effort` or `acp`, an effort outside the five levels,
|
|
a `max_concurrent_reviews` that isn't a whole number, a runtime other than
|
|
the two, or an `acp` without `base_url`, `model` or `prices`. Also **400**
|
|
when the document the PUT would store breaks a rule: an empty `model`, one
|
|
with whitespace or one starting with `-`, a `max_concurrent_reviews` outside
|
|
1 to 8, an `acp.base_url` that isn't `https://`, an empty `acp.model` or one
|
|
with whitespace, a negative price, or the `acp` runtime with no `acp`
|
|
- **401** for a PUT with no `Remote-User`
|
|
- **503** when this host has no swarm queue, or it isn't connected
|
|
- **500** when the controller can't read or write the settings
|
|
|
|
When the controller can't read the settings, a try-out answers the same 503
|
|
or 500, and a review posts a no-verdict comment. The operator writes each
|
|
runtime's key by hand in the secret store
|
|
([`services.md`](services.md#review-classifier)).
|
|
|
|
The per-agent equivalent is [`inference.md`](inference.md), where the
|
|
controller writes model and effort to a queue KV bucket
|
|
(`swarm-queue-client/src/agent_inference.rs`) and each agent reads its own
|
|
key.
|
|
|
|
### The service account
|
|
|
|
A dedicated forge user, separate from swarm-controller's site-admin account
|
|
and from every agent's account. It's the reviewer someone requests, it
|
|
authors every classifier review, and it pushes the branches the try-out section
|
|
creates.
|
|
|
|
`swarm-controller` creates it as `review-classifier`, with the email
|
|
`review-classifier@services.hyperhive.local` and no repo creation, and mints
|
|
its one token, `swarm-review-classifier` with scope `write:repository`, into
|
|
the store at `swarm/services/review-classifier/forge/token`. A pass at start
|
|
and every five minutes re-mints when the token is missing or no longer
|
|
matches the forge. The controller refuses an existing account by that name
|
|
with any other email. The controller posts the reviews and pushes the try-out
|
|
branches with that token, and the classifier service never sees it.
|
|
|
|
**Giving a repo access is a repo admin's step, by hand.** Nothing grants
|
|
it automatically. In the repo's settings, add `review-classifier` as a
|
|
collaborator:
|
|
|
|
- with **read**, so someone can request it as a reviewer, and its reviews show,
|
|
but its approval doesn't count
|
|
- with **write**, so its approval counts toward the required approvals and
|
|
its request-changes blocks the merge where the branch protection blocks on
|
|
rejected reviews. Where the branch protection restricts approvals to an
|
|
allowlist, the account counts only once it's on that list, and read access
|
|
is then enough for reviews.
|
|
|
|
The try-out's **open a PR** pushes a branch to the repo itself, so it
|
|
needs **write** whatever the branch protection says.
|
|
|
|
The repo also needs a rules file on its default branch (see
|
|
[Repo configuration](#repo-configuration)). Without one, a request posts a
|
|
comment saying so.
|
|
|
|
## The classification run
|
|
|
|
### Shared context
|
|
|
|
Every classification in a run sees the same context, in the same order:
|
|
|
|
1. a short system prompt
|
|
2. the PR's commits, each with its full message
|
|
3. the full diff
|
|
|
|
Only the last part differs per classification: that classification's
|
|
description and the task to classify the PR against it. The identical prefix
|
|
lets the runtime's prompt cache reuse it across questions.
|
|
|
|
On the API key runtime the commits and diff are one content block of the user
|
|
message and the question another, with a cache breakpoint at the end of the
|
|
first. The cache writes an entry only where a breakpoint is, and `claude` sets
|
|
one only after the last block, so that breakpoint is what lets the next
|
|
question read the commits and diff from the cache. The PR stays in the user
|
|
message rather than the system prompt, which tells the model to judge it as
|
|
data. A shared context shorter than the model's minimum cacheable length is
|
|
not cached at all.
|
|
|
|
No object hash is in that prefix: the header names the repo and the PR number,
|
|
not the head commit, each commit's heading gives its position instead of its
|
|
sha, and the diff loses each file's `index <hash>..<hash>` line, keeping its
|
|
`diff --git` paths and every hunk. A rebase rewrites all of those hashes, so
|
|
with them in, a rebased PR with the same messages and hunks would miss the
|
|
cache from the first line. The head commit still goes to the controller,
|
|
which posts the review against it.
|
|
|
|
### One question, one answer
|
|
|
|
The model answers each classification in JSON. The answer holds:
|
|
|
|
- a **verdict**, one of _match_, _no match_ or _not applicable to this PR_
|
|
- a **certainty** from 0 to 100, how sure the model is of that verdict
|
|
- a **reason**; the prompt asks for under 350 characters, and the classifier
|
|
truncates one over 500 to 499 plus `…`, keeping the verdict and certainty
|
|
|
|
An answer:
|
|
|
|
```json
|
|
{
|
|
"verdict": "no-match",
|
|
"certainty": 92,
|
|
"reason": "Two new timestamps use naive datetimes without a zone."
|
|
}
|
|
```
|
|
|
|
The verdict field holds `match`, `no-match` or `not-applicable`, and the
|
|
certainty field a whole number. A runtime error, a timeout, or an answer that
|
|
isn't that JSON object leaves the classification without a verdict: it's listed
|
|
with its error, and doesn't count. A run with no key for its runtime, or in which no
|
|
classification got an answer, fails as a whole, and shows its error instead of a
|
|
verdict.
|
|
|
|
### Aggregation into the verdict
|
|
|
|
The classifier collects the answers into one result, keyed by
|
|
classification name, and logs it. A classification counts toward the
|
|
overall verdict only if it's applicable and its certainty reaches the
|
|
threshold. The table still lists every classification below its threshold or
|
|
not applicable, marked as excluded.
|
|
|
|
A counted classification fails when its verdict lands on the wrong side of
|
|
its polarity: _no match_ on a positive one, _match_ on a negative one. Any
|
|
failing classification makes the verdict **request changes**. Otherwise the
|
|
verdict is **approve**, including when the run counted no classification at
|
|
all.
|
|
|
|
### The review
|
|
|
|
The service account posts the verdict as a forge review, approve or request
|
|
changes, and the table is the review's body. An illustrative table:
|
|
|
|
| result | check | certainty / threshold | reason |
|
|
| ----------- | ------------ | --------------------- | -------------------------------------------- |
|
|
| ❌ no match | datetime | 92 / 80 | Two new timestamps use naive datetimes. |
|
|
| ✅ no match | no-ads | 99 / 90 | No commit message carries an AI attribution. |
|
|
| ❔ unsure | bao-first | 95 / 70 | The PR adds no secret. |
|
|
| ❔ match | docs-updated | 41 / 60 | Below threshold. |
|
|
|
|
**Overall: ❌ request changes**
|
|
|
|
The result cell is the symbol for what the row did to the review, then the
|
|
model's verdict. ✅ is a counted row that passed its polarity and ❌ a counted
|
|
row that failed it, so _match_ is ✅ on a positive rule and ❌ on a negative one.
|
|
❔ is a row that doesn't count: below its threshold, not applicable (shown as
|
|
_unsure_), or without an answer. The certainty cell reads _certainty / threshold_.
|
|
|
|
The approval is a real one: it counts toward a repo's required approvals
|
|
like any other ([`pr-review-gate.md`](../process/pr-review-gate.md)). The
|
|
Repos that rely on it typically raise their required approval count to
|
|
account for it.
|
|
|
|
A run with no clean verdict never approves. When the PR or the rules don't
|
|
load, the run fails, the classifier answers a different number of
|
|
classifications than the controller sent it, or **any** classification has no answer,
|
|
the account posts a **comment** instead, starting with `**No verdict:**` and the reason,
|
|
followed by the table when there is one. A comment neither approves nor
|
|
blocks the merge: nine timeouts and one pass post a comment, not an
|
|
approval.
|
|
|
|
The controller flattens every cell to one line, with `|`, backticks, brackets, `<`, `>`,
|
|
`&`, `#` and other markup characters escaped, and a zero-width space after
|
|
each `@`, `#`, `!`, `://` and `www.`, so a model's `reason` can't break the
|
|
table, embed markup, mention anyone, reference an issue or PR, or carry a
|
|
link.
|
|
|
|
## Runtimes
|
|
|
|
The classifier runs on one of two runtimes, the settings' `runtime`:
|
|
|
|
- **API key** (`api-key`, the default): the `claude` CLI on the Anthropic API
|
|
key at `secret/swarm/services/review-classifier/anthropic-api-key`, at
|
|
`model` and `effort`
|
|
- **ACP** (`acp`): `opencode` on the provider `acp` names, with the provider
|
|
key at `secret/swarm/services/review-classifier/acp-provider`, at
|
|
`acp.model`
|
|
|
|
Model names don't carry across runtimes. A claude alias means nothing to an
|
|
ACP provider, so each runtime has a model field of its own.
|
|
|
|
To switch to ACP, an admin:
|
|
|
|
1. writes the provider key in the secret store's UI under the OIDC login, at
|
|
`secret/swarm/services/review-classifier/acp-provider`, field `api_key`
|
|
2. PUTs the runtime and the provider:
|
|
|
|
```json
|
|
{
|
|
"runtime": "acp",
|
|
"acp": {
|
|
"base_url": "https://llm.example/v1",
|
|
"model": "Qwen/Qwen3-35B",
|
|
"prices": { "input": 0.5, "output": 2 }
|
|
}
|
|
}
|
|
```
|
|
|
|
The next run goes to that provider. `{"runtime": "api-key"}` switches back and
|
|
keeps the provider for next time.
|
|
|
|
A split deploy runs on the API key: a classifier older than the ACP runtime
|
|
ignores `runtime` and runs every classification on `claude`, and a controller
|
|
older than it ignores the stored `runtime` and `acp` and sends neither.
|
|
|
|
A third runtime, **claude subscription** (the `claude` CLI on a Claude
|
|
subscription login), comes later.
|
|
|
|
> **Open point.** The subscription runtime needs a login, which is the one
|
|
> piece of durable state the service would keep. A swarm-level claude
|
|
> subscription login may have to exist before this runtime works.
|
|
|
|
## Swarm UI
|
|
|
|
Two pages in the [swarm UI](ui.md), `admins` only like the rest of it.
|
|
|
|
### Recent runs
|
|
|
|
One card per run: the repo, the PR, and how many classifications ran.
|
|
Selecting a card expands it to show each classification's result, the same
|
|
rows as the review table.
|
|
|
|
### Try-out
|
|
|
|
A page for writing classification prompts without touching the repo. Pick a
|
|
repo and a PR, write or edit a classification, and run it against that PR;
|
|
change the prompt and run it again until it answers the way you want.
|
|
Nothing reaches the forge while you iterate.
|
|
|
|
The page posts the run to `swarm-controller` (`POST
|
|
/api/review-classifier/try-out`), which reads the PR with its own forge token
|
|
and sends it to the classifier as one queue request on
|
|
`$SWARM.review-classifier.try-out`. The page then reads the result back (`GET
|
|
/api/review-classifier/try-out/{id}`): a run can take longer than a request
|
|
stays open. The controller keeps the result in memory for an hour, and the
|
|
finished run goes into the [run store](#run-store-and-retention).
|
|
|
|
`GET /api/review-classifier/rules?repo=owner/name` returns the repo's own
|
|
rules, for the page to load into its editor; a run posts them like typed
|
|
ones. The route answers:
|
|
|
|
- **200** with `path` (the file read), `branch` (the default branch),
|
|
`commit` (the commit the route read the file at) and `classifications`, in
|
|
the file's order and in the POST's shape
|
|
- **400** for a `repo` that's missing or not `owner/name`
|
|
- **404** for a repo the forge doesn't know; a default branch with no commit,
|
|
in an empty repo or when the default branch doesn't exist; or a default
|
|
branch with neither file
|
|
- **422** for both files on the default branch, or a file that doesn't parse
|
|
or breaks a rule from [Repo configuration](#repo-configuration); the
|
|
`detail` names the file and what's wrong
|
|
- **503** when this host has no forge token yet, or the forge failed or
|
|
didn't answer within two minutes
|
|
|
|
The page's repo and PR fields offer values to pick from, and stay free text,
|
|
since a run also works on a merged or closed PR. Two routes feed them:
|
|
|
|
- `GET /api/pulls/repos` lists the repos with at least one open PR, as
|
|
`owner/name`.
|
|
- `GET /api/pulls?repo=owner/name` lists the repo's open PRs, highest number
|
|
first, each with `number`, `title` and `head_sha`. It answers **400** for a
|
|
`repo` that's missing or not `owner/name`, **404** for a repo the forge
|
|
doesn't know, and **503** when this host has no forge token yet, or the
|
|
forge failed or didn't answer within two minutes.
|
|
|
|
Every error is problem+json with the reason in `detail`.
|
|
|
|
When it's ready, **open a PR** (`POST /api/review-classifier/open-pr`, with
|
|
the repo and the classifications) has the service account push a branch that
|
|
adds the classifications to the repo's rules, with you, the swarm UI user, as
|
|
commit author. The page then links you to the forge's compare page for that
|
|
branch, where you log in as usual and open the PR yourself.
|
|
|
|
- **The branch** is `review-classifier/<you>-<8 hex digits>`, made from the
|
|
default branch's head with one commit, through the forge's contents API.
|
|
- **The file** is the rules file on the default branch, merged by name: a
|
|
classification named like an entry replaces it, a new name goes at the end,
|
|
and every other entry stays. A repo with no rules file gets
|
|
`.hyperhive/review-classifier.yml`. In a YAML file, the entries the merge
|
|
leaves alone keep their text, comments included. The merged file has to
|
|
pass the same checks the `rules` route makes.
|
|
- **The author** is the forge account named like your swarm UI login. A forge
|
|
account made by an SSO login carries the SSO username, so it's yours once
|
|
you've logged in to the forge. The commit carries its full name, or the
|
|
login when it has none, and its noreply address,
|
|
`<login, lowercased>@noreply.<forge domain>`. It's the address the forge
|
|
itself uses for an account that keeps its email private, so the commit
|
|
links to your account and your email stays out of the repo's history. The
|
|
committer is the service account.
|
|
- **The token** is the service account's own, read from the store for each
|
|
push (see [The service account](#the-service-account)), so the account
|
|
needs **write** access to the repo.
|
|
|
|
The route answers **201** with `branch`, `path` (the file written) and
|
|
`compare_url`, or:
|
|
|
|
- **400** for a body the try-out POST would refuse in its repo or
|
|
classifications
|
|
- **401** without the `Remote-User` header the gateway sets
|
|
- **403** when the forge refuses the push: the service account isn't a
|
|
collaborator with write access
|
|
- **404** for a repo the forge doesn't know, or one with no commit on its
|
|
default branch
|
|
- **409** when the forge has no account named like you yet, or the rules
|
|
file changed on the default branch during the push
|
|
- **422** for a default branch holding both rules files or one the `rules`
|
|
route refuses; classifications the file already holds unchanged; or a
|
|
merge that breaks a rule, such as more than ten classifications
|
|
- **503** when this host has no forge token, the service account's token
|
|
isn't minted yet, or the forge failed or didn't answer within two minutes
|
|
|
|
## Run store and retention
|
|
|
|
`swarm-controller` records every run when it ends: a try-out when the
|
|
classifier's reply arrives, a requested review once the controller has posted
|
|
its review or the post has failed, and a request turned away because the
|
|
queue was full once its comment is on the forge or the post has failed. A review request for a head that's already
|
|
running or waiting starts nothing and isn't recorded. A run still going isn't
|
|
in the list, and a controller restart drops a run in progress.
|
|
|
|
The newest 100 runs are one JSON array under the key `recent` in the
|
|
`review-classifier-runs` KV bucket on the swarm queue, which only the
|
|
controller reads or writes. The controller creates the bucket on first use and
|
|
adds each run with a compare-and-set write, so the list survives a controller
|
|
restart or a move to another host. Recording the 101st run drops the oldest;
|
|
there's no time-based retention. The controller leaves a run it can't decode, such
|
|
as one a newer build recorded, out of the list with a warning in its log. It
|
|
keeps the run in the store as it was and counts it toward the 100. A document
|
|
that isn't a list reads as no runs. The controller logs a failed write, and it
|
|
changes nothing on the forge.
|
|
|
|
`GET /api/review-classifier/runs` returns the list, newest first. Like the
|
|
try-out GET, the gateway's `admins` rule gates it, and the controller checks no
|
|
header. Each run has:
|
|
|
|
- `id`: a try-out's is the id its POST answered with
|
|
- `trigger`: `try-out` or `review`
|
|
- `repo` (`owner/name`), `pr`, and `requester`: the swarm UI user who started
|
|
the try-out, or the forge user who requested the review
|
|
- `head_sha`: the commit the run read, absent when the PR wasn't read, as for
|
|
an unreadable PR or a full queue
|
|
- `started_unix` and `finished_unix`: a try-out starts at its POST, a review
|
|
once it has a run place
|
|
- `outcome`: `approved`, `changes-requested`, `no-verdict`, `queue-full` or
|
|
`post-failed` for a review; `done` or `failed` for a try-out
|
|
- `detail`: the reason, for every outcome but `approved`,
|
|
`changes-requested` and `done`. For `post-failed` it's the run's own reason,
|
|
when it has one, then why the post failed.
|
|
- `rows` and `overall`, as the try-out GET returns them, when the classifier
|
|
answered
|
|
- `review_url`: the posted review, when the forge names its URL
|
|
|
|
A run holds no diff, no commit text and no token count or cost. The route
|
|
answers 503 when the controller has no swarm queue or can't reach the bucket,
|
|
and 500 when it can't read the list, both as problem+json.
|
|
|
|
Each recorded run is also one `info` event from `swarm-controller` with the
|
|
message `review classifier run finished`. The event carries the same fields,
|
|
with `rows` as JSON. It reaches VictoriaLogs through the host journal, where
|
|
each field arrives as `F_` and the name in capitals (`F_OUTCOME`, `F_REPO`), so
|
|
a run whose write failed is still there:
|
|
|
|
```console
|
|
$ swarm-logs query '_time:7d _SYSTEMD_UNIT:="swarm-controller.service" _msg:="review classifier run finished" | fields _time, F_TRIGGER, F_REPO, F_PR, F_OUTCOME, F_DETAIL, F_REVIEW_URL' --format json
|
|
```
|
|
|
|
Add a filter such as `F_OUTCOME:="post-failed"` before the `|` to narrow it.
|
|
|
|
## Observability and cost
|
|
|
|
The swarm UI shows no token counts, cache counts or cost, on any page. On the
|
|
claude runtimes, every classifier run starts its `claude` process with
|
|
`OTEL_RESOURCE_ATTRIBUTES` naming the repo and the PR, so the cost reads from
|
|
the metrics store by repo and PR. The process exports to the hive-tier collector on
|
|
the classifier's host, where that host runs one. On a subscription, cost is notional.
|
|
|
|
On the ACP runtime, the service computes each classification's cost itself,
|
|
from the settings' `acp.prices` and the token counts the agent reports for the
|
|
prompt, and adds it to the counter `hyperhive.review_classifier.cost.usage`,
|
|
labelled `repo`, `pr`, `model` (the provider's) and `currency` (`USD`). It
|
|
exports to the same collector `claude` does. A classification stopped at its
|
|
deadline, or whose agent reports no token counts, adds nothing. opencode
|
|
reports no cost of its own for a provider without prices in its config, which
|
|
is why the prices are a setting.
|
|
|
|
Each such classification also adds the tokens it used to the counter
|
|
`hyperhive.review_classifier.token.usage`, with the same labels bar `currency`
|
|
and a `type` of `input`, `output`, `cacheRead` or `cacheCreation`. Those are
|
|
the types of `claude_code.token.usage`, which `claude` exports on the claude
|
|
runtimes, so the cache hit rate reads the same way on every runtime: `cacheRead`
|
|
over the sum of `input`, `cacheRead` and `cacheCreation`.
|
|
|
|
The classifier logs each run's result to its journal, which ships to
|
|
VictoriaLogs like every swarm service container's
|
|
([`services.md`](services.md#logs-victorialogs)).
|
|
|
|
## Still open
|
|
|
|
These get settled, and filled in here, as the work reaches them:
|
|
|
|
- **Cache strategy.** Fully independent sessions per classification, or one
|
|
warm session and then the rest in parallel. To measure through the cache
|
|
token counts before choosing.
|
|
- **Oversized diffs.** What happens when the diff doesn't fit the model's
|
|
context. Lockfiles already collapse to one line each, as in `hive-forge diff`.
|
|
- **Budget.** Any per-run or per-day spend limit on top of the concurrency
|
|
cap.
|
|
- **Diff-leaves-swarm policy.** On the API key and ACP runtimes the diff
|
|
goes to a third-party provider. Which repos may use which runtime, and
|
|
whether anything gets redacted.
|
|
- **Prompt injection.** A PR's diff and commit messages are model input, so
|
|
a PR can try to talk the classifier into approving it.
|
|
- **Subscription credential.** Where in bao the subscription runtime's login
|
|
lives. The service account's forge token is at
|
|
`secret/swarm/services/review-classifier/forge/token`.
|
|
- **Reviewer handoff.** How a later reviewer, human or agent, learns the
|
|
classifier passed a PR before it starts its own review.
|