docs: review classifier architecture (planned)
Add docs/swarm/review-classifier.md, the high-level architecture of the planned PR review classifier pipeline, written as the full target shape under a "planned, not implemented yet" banner. Link it from docs/README.md and the docs/swarm/README.md piece list. The page reflects the epic description and the operator's answers in the epic thread; points still open are listed in its "Still open" section. The banner names the epic by title rather than by number because the tracker-tag lint rejects issue references in tracked files.
This commit is contained in:
atlas
parent
87aeed32a7
commit
c0af8d1fd3
3 changed files with 301 additions and 0 deletions
|
|
@ -86,6 +86,9 @@ declarations.
|
|||
twice after downtime?** → [`swarm/schedules.md`](swarm/schedules.md).
|
||||
- **How do I set an agent's model or effort from the swarm, and what happens
|
||||
to its own pick?** → [`swarm/inference.md`](swarm/inference.md).
|
||||
- **How will a PR get checked against its repo's own yes/no rules before
|
||||
argus reviews it?** → [`swarm/review-classifier.md`](swarm/review-classifier.md)
|
||||
(planned, not implemented yet).
|
||||
|
||||
## Scheduler, CI, observability
|
||||
|
||||
|
|
|
|||
|
|
@ -25,6 +25,9 @@ grafana, the controller, authelia, …) →
|
|||
kept by the swarm and fired in each agent's container. → [`schedules.md`](schedules.md)
|
||||
- **inference settings** — the model and effort the swarm sets for an agent,
|
||||
over its own. → [`inference.md`](inference.md)
|
||||
- **review classifier** — planned, not implemented yet: checks a PR against
|
||||
its repo's own yes/no rules and posts the result as a review. →
|
||||
[`review-classifier.md`](review-classifier.md)
|
||||
- **swarm UI** — the operator's day-to-day surface, on the swarm apex,
|
||||
`admins` only. → [`ui.md`](ui.md)
|
||||
- **shared services** — one forge, homeserver, SSO, queue and metrics/logs
|
||||
|
|
|
|||
295
docs/swarm/review-classifier.md
Normal file
295
docs/swarm/review-classifier.md
Normal file
|
|
@ -0,0 +1,295 @@
|
|||
# Review classifier
|
||||
|
||||
> ⚠️ **PLANNED, NOT IMPLEMENTED YET.** This page describes the full target
|
||||
> shape of the review classifier at component level, and none of it exists
|
||||
> in the tree today. The forge epic "pr review classifier pipeline" tracks
|
||||
> the work. Details land here as each part ships, and this banner goes away
|
||||
> when the epic closes.
|
||||
|
||||
The review classifier checks a pull request against short, named yes/no
|
||||
rules that the repo itself defines, and posts the result as a forge review.
|
||||
A small model answers one rule at a time, so mundane rule checks come out
|
||||
the same way on every PR.
|
||||
|
||||
It doesn't replace argus's review. argus reviews a PR only once the
|
||||
classifier has passed it, and argus runs on a bigger model. The classifier
|
||||
takes the cheap, repetitive checks off argus.
|
||||
|
||||
## Repo configuration
|
||||
|
||||
A repo opts in with a config file in a `.hyperhive/` directory at its root,
|
||||
in YAML or JSON. The classifier always reads it from the repo's **default
|
||||
branch**, never from the PR, so a PR can't edit the rules it's judged by.
|
||||
|
||||
Each classification has:
|
||||
|
||||
- a **name**, short and unique in the repo, shown in the result table
|
||||
- a **description**: the yes/no question the model answers about the PR
|
||||
- a **certainty threshold** from 0 to 100
|
||||
- a **polarity**: _positive_ means a match is the wanted outcome, and
|
||||
_negative_ means a match is a violation
|
||||
|
||||
An illustrative example. The file name and every field name in it are
|
||||
placeholders, not a fixed schema:
|
||||
|
||||
```yaml
|
||||
# illustrative only: file name and field names aren't fixed
|
||||
classifications:
|
||||
- name: datetime
|
||||
description: All datetimes carry time zone information.
|
||||
polarity: positive
|
||||
threshold: 80
|
||||
- name: no-ads
|
||||
description: >-
|
||||
A commit message contains a co-authored-by line or other
|
||||
advertising for an AI model.
|
||||
polarity: negative
|
||||
threshold: 90
|
||||
- name: bao-first
|
||||
description: Any added secret reaches its reader through the bao store.
|
||||
polarity: positive
|
||||
threshold: 70
|
||||
- name: docs-updated
|
||||
description: >-
|
||||
The PR updates the operator-facing docs where the code change
|
||||
needs it.
|
||||
polarity: positive
|
||||
threshold: 60
|
||||
```
|
||||
|
||||
## Trigger and lifecycle
|
||||
|
||||
The classifier has a dedicated forge service account. Requesting that
|
||||
account as a reviewer on a PR starts a run:
|
||||
|
||||
1. The forge sends the review-request webhook to
|
||||
[`swarm-controller`](../../swarm-controller/README.md), the swarm's one
|
||||
webhook owner.
|
||||
2. The controller hands the run to the classifier service as an async
|
||||
message.
|
||||
3. The classifier runs every classification in the repo's config, read from
|
||||
the default branch, against the PR's commits and diff.
|
||||
4. The service account posts the result on the PR as a review.
|
||||
|
||||
A verdict belongs to the commit it ran on. Once the PR's head moves,
|
||||
requesting the service account as a reviewer again starts a fresh run on the
|
||||
new head.
|
||||
|
||||
That only holds if the bot's review on an older commit stops counting once
|
||||
the head moves. Either the repo enables Forgejo's stale-review dismissal
|
||||
(see [PR review gate](../process/pr-review-gate.md)), or the service
|
||||
dismisses its own earlier review when it posts a new one. Where neither
|
||||
applies, an old approval keeps counting and an old request-changes keeps
|
||||
blocking.
|
||||
|
||||
Runs go in parallel, up to a maximum set at swarm level.
|
||||
|
||||
## Components
|
||||
|
||||
### The classifier service
|
||||
|
||||
A swarm-level service in its own container, on whichever host the operator
|
||||
puts it. It talks to `swarm-controller` through async messages only, and
|
||||
gets its secrets from the swarm's secret store, bao
|
||||
([`bao.md`](bao.md), [`credentials.md`](credentials.md)).
|
||||
|
||||
It looks like an agent from the inside, with three things missing: it has no
|
||||
full agent identity, no state directory and no tools. Every model session is
|
||||
ephemeral, and nothing carries over from one run to the next.
|
||||
|
||||
The model sessions run through [`hive-runtime`](../../hive-runtime/README.md),
|
||||
the crate `hive-agent` and `hive-subagent-mcp` already drive their turns
|
||||
through: the `Runtime` trait in `hive-runtime/src/lib.rs`, with
|
||||
`ClaudeRuntime` and `AcpRuntime` behind it. That trait runs turns on a durable
|
||||
session, and the subagent daemon's claude path
|
||||
(`hive-subagent-mcp/src/session.rs`) spawns claude directly. A refactor
|
||||
gives `hive-runtime` a one-off, cancellable session shape first, so the
|
||||
classifier, the agent and the subagent daemon share one runtime
|
||||
implementation.
|
||||
|
||||
### Swarm-level settings
|
||||
|
||||
The operator sets the classifier's settings for the whole swarm:
|
||||
|
||||
- the runtime (see [Runtimes](#runtimes)) and its credentials
|
||||
- the model and effort, and related inference settings
|
||||
- the maximum number of concurrent runs
|
||||
|
||||
The per-agent equivalent is [`inference.md`](inference.md), where the
|
||||
controller writes model and effort to a queue KV bucket
|
||||
(`swarm-queue-client/src/agent_inference.rs`) and each agent reads its own
|
||||
key.
|
||||
|
||||
### The service account
|
||||
|
||||
A dedicated forge user, separate from swarm-controller's site-admin account
|
||||
and from every agent's account. It's the reviewer someone requests, it
|
||||
authors every classifier review, and it pushes the branches the try-out page
|
||||
creates.
|
||||
|
||||
## The classification run
|
||||
|
||||
### Shared context
|
||||
|
||||
Every classification in a run sees the same context, in the same order:
|
||||
|
||||
1. a short system prompt
|
||||
2. the PR's commits, each with its full message
|
||||
3. the full diff
|
||||
|
||||
Only the last part differs per classification: that classification's
|
||||
description and the task to classify the PR against it. The identical prefix
|
||||
lets the runtime's prompt cache reuse it across questions.
|
||||
|
||||
### One question, one answer
|
||||
|
||||
The model answers each classification in JSON. The answer holds:
|
||||
|
||||
- a **verdict**, one of _match_, _no match_ or _not applicable to this PR_
|
||||
- a **certainty** from 0 to 100, how sure the model is of that verdict
|
||||
- a **reason**, limited in length
|
||||
|
||||
An illustrative answer, with placeholder field names:
|
||||
|
||||
```json
|
||||
{
|
||||
"verdict": "no-match",
|
||||
"certainty": 92,
|
||||
"reason": "Two new timestamps use naive datetimes without a zone."
|
||||
}
|
||||
```
|
||||
|
||||
### Aggregation into the verdict
|
||||
|
||||
The classifier collects the answers into one result, keyed by
|
||||
classification name, and logs it. A classification counts toward the
|
||||
overall verdict only if it's applicable and its certainty reaches the
|
||||
threshold. The table still lists every classification below its threshold or
|
||||
not applicable, marked as excluded.
|
||||
|
||||
A counted classification fails when its verdict lands on the wrong side of
|
||||
its polarity: _no match_ on a positive one, _match_ on a negative one. Any
|
||||
failing classification makes the verdict **request changes**. Otherwise the
|
||||
verdict is **approve**, including when the run counted no classification at
|
||||
all.
|
||||
|
||||
### The review
|
||||
|
||||
The service account posts the verdict as a forge review, approve or request
|
||||
changes, and the table is the review's body. An illustrative table:
|
||||
|
||||
| classification | polarity | verdict | certainty | threshold | counted | reason |
|
||||
| -------------- | -------- | -------------- | --------- | --------- | ------- | -------------------------------------------- |
|
||||
| datetime | positive | no match | 92 | 80 | yes ❌ | Two new timestamps use naive datetimes. |
|
||||
| no-ads | negative | no match | 99 | 90 | yes ✅ | No commit message carries an AI attribution. |
|
||||
| bao-first | positive | not applicable | 95 | 70 | no | The PR adds no secret. |
|
||||
| docs-updated | positive | match | 41 | 60 | no | Below threshold. |
|
||||
|
||||
**Overall: request changes.**
|
||||
|
||||
The approval is a real one: it counts toward a repo's required approvals
|
||||
like any other ([`pr-review-gate.md`](../process/pr-review-gate.md)). The
|
||||
hyperhive repo requires more approvals to account for it.
|
||||
|
||||
## Runtimes
|
||||
|
||||
The classifier runs on one of three runtimes, chosen at swarm level:
|
||||
|
||||
- **claude subscription**: the `claude` CLI on a Claude subscription login
|
||||
- **API key**: the `claude` CLI on an API key
|
||||
- **ACP**: an ACP agent and its provider, as for an ACP agent today
|
||||
|
||||
Model names don't carry across runtimes. A claude alias means nothing to an
|
||||
ACP provider, so the model setting belongs to the runtime it names.
|
||||
|
||||
> **Open point.** The subscription runtime needs a login, which is the one
|
||||
> piece of durable state the service would keep. A swarm-level claude
|
||||
> subscription login may have to exist before this runtime works.
|
||||
|
||||
## Swarm UI
|
||||
|
||||
Two pages in the [swarm UI](ui.md), `admins` only like the rest of it.
|
||||
|
||||
### Recent runs
|
||||
|
||||
One card per run: the repo, the PR, how many classifications ran, and the
|
||||
run's overall cost. Selecting a card expands it to show each
|
||||
classification's result, the same rows as the review table.
|
||||
|
||||
### Try-out
|
||||
|
||||
A page for writing classification prompts without touching the repo. Pick a
|
||||
repo and a PR, write or edit a classification, and run it against that PR;
|
||||
change the prompt and run it again until it answers the way you want.
|
||||
Nothing reaches the forge while you iterate.
|
||||
|
||||
When it's ready, **open a PR** has the service account push a branch that
|
||||
adds the classification to the repo's config, with you, the swarm UI user,
|
||||
as commit author. The page then links you to the forge's new-PR page for
|
||||
that branch, where you log in as usual and open the PR yourself.
|
||||
|
||||
## Observability and cost
|
||||
|
||||
Every run records its cost, which the recent-runs page shows. Where the cost
|
||||
comes from depends on the runtime:
|
||||
|
||||
- An ACP agent reports a running cost total (`ReportedCost` in
|
||||
`hive-runtime/src/acp/stream.rs`); a session's last total is its cost.
|
||||
- The `claude` CLI reports token counts, including cache reads and writes.
|
||||
The dashboard already estimates a cost from token counts with a price
|
||||
table (`modelPrices` in [`dashboard.md`](../web-ui/dashboard.md)).
|
||||
- On a subscription, cost is notional.
|
||||
|
||||
The cache token counts also show whether the shared context gets the cache
|
||||
reuse it's meant to.
|
||||
|
||||
The classifier logs each run's result to its journal, which ships to
|
||||
VictoriaLogs like every swarm service container's
|
||||
([`services.md`](services.md#logs-victorialogs)).
|
||||
|
||||
## Still open
|
||||
|
||||
These get settled, and filled in here, as the work reaches them:
|
||||
|
||||
- **Cache strategy.** Fully independent sessions per classification, or one
|
||||
warm session and then the rest in parallel. To measure through the cache
|
||||
token counts before choosing.
|
||||
- **Run store and retention.** Where run records live, how long they stay,
|
||||
and how the recent-runs page reads them.
|
||||
- **Failure handling.** What a runtime error, malformed JSON, a timeout or a
|
||||
rate limit does to a classification and to the verdict.
|
||||
- **JSON validation and the reason cap.** The exact answer schema, and the
|
||||
reason's length limit.
|
||||
- **Oversized diffs.** What happens when the diff doesn't fit the model's
|
||||
context, and whether lockfiles collapse as in `hive-forge diff`
|
||||
(`hive-forge/src/verbs/diff.rs`).
|
||||
- **Stale reviews.** Whether the repo's stale-review dismissal setting
|
||||
retires the bot's review on an older commit, or the service dismisses its
|
||||
own earlier review when it posts a new one.
|
||||
- **Budget.** Any per-run or per-day spend limit on top of the concurrency
|
||||
cap.
|
||||
- **Diff-leaves-swarm policy.** On the API key and ACP runtimes the diff
|
||||
goes to a third-party provider. Which repos may use which runtime, and
|
||||
whether anything gets redacted.
|
||||
- **Prompt injection.** A PR's diff and commit messages are model input, so
|
||||
a PR can try to talk the classifier into approving it.
|
||||
- **The "no tools" mechanism.** How each runtime runs a session with no
|
||||
tools at all.
|
||||
- **Credential paths.** Where in bao the runtime credentials and the service
|
||||
account's forge token live, and which component holds the token.
|
||||
- **Model naming across runtimes.** How the swarm-level model setting
|
||||
handles runtime-specific model names.
|
||||
- **Cost method per runtime and telemetry labels.** The exact cost source
|
||||
per runtime, and the resource attributes that keep the classifier's spend
|
||||
separate from the agents' on the existing dashboards.
|
||||
- **Async message medium.** Which queue subjects or streams carry the
|
||||
messages between `swarm-controller` and the classifier.
|
||||
- **argus handoff.** How argus learns the classifier passed a PR before it
|
||||
starts its own review.
|
||||
- **Opt-in details.** Beyond the config file: the service account's access
|
||||
to a repo, so someone can request it as reviewer and it can push try-out
|
||||
branches.
|
||||
- **Commit author for try-out PRs.** How the classifier derives the commit
|
||||
author's name and email from the swarm UI login.
|
||||
- **Refactor scope.** How much of `hive-agent`'s and `hive-subagent-mcp`'s
|
||||
runtime code moves into `hive-runtime` before the classifier uses it.
|
||||
Loading…
Reference in a new issue