review classifier: export request, verdict, check and run metrics, with a dashboard
The controller counts review requests, verdicts and per-check results and times each run; the classifier service counts retried asks. A provisioned Grafana dashboard charts them.
This commit is contained in:
parent
7680b07fcb
commit
eaf26aaadd
13 changed files with 1390 additions and 28 deletions
|
|
@ -522,6 +522,38 @@ at warn, with the kind of failure (`token`, `transport` or `status <code>`)
|
|||
and the start of the response body. It repeats a persisting failure every
|
||||
10 minutes with a count, and logs once at info when pushes succeed again.
|
||||
|
||||
## Review classifier result metrics (`swarm-controller`)
|
||||
|
||||
`swarm-controller` counts what review requests and review runs come to, from
|
||||
the same exporter as the VCS metrics above. The classifier service adds the
|
||||
retry counter beside its cost counters
|
||||
([`review-classifier.md`](../swarm/review-classifier.md#observability-and-cost)).
|
||||
|
||||
| Metric | Unit | Kind | Attributes |
|
||||
| ------------------------------------------ | ---- | --------- | ------------------------------------------------------------------------------------ |
|
||||
| `hyperhive.review_classifier.requests` | — | counter | `outcome`: `queued`, `queue_failed`, `unparsable`, `deferred`, `started`, `reposted` |
|
||||
| `hyperhive.review_classifier.verdicts` | — | counter | `outcome`: `approve`, `request_changes`, `no_verdict` |
|
||||
| `hyperhive.review_classifier.checks` | — | counter | `check` (the rule's name), `result`: `match`, `no_match`, `unsure`, `no_answer` |
|
||||
| `hyperhive.review_classifier.run.duration` | `s` | histogram | `outcome`: the verdict names, or `post_failed` |
|
||||
| `hyperhive.review_classifier.ask.retries` | — | counter | `runtime`: `api_key`, `acp` |
|
||||
|
||||
A verdict means the classifier decided, before it posts the review: a run
|
||||
whose post fails still adds to `verdicts`, and shows as `post_failed` in
|
||||
`run.duration`. A repo with no rules file gets an approval with no model call and
|
||||
counts as an `approve` verdict and run, with no `checks`.
|
||||
|
||||
`deferred` counts each time the queue hands over a request for a PR whose
|
||||
review is still going, so a long review adds one per retry wait. `reposted`
|
||||
is a request for a head already reviewed: it posts the earlier verdict and
|
||||
adds to `verdicts` but runs no checks, so it adds nothing to `checks` or
|
||||
`run.duration`. A request the queue refuses posts a comment and counts
|
||||
as `queue_failed` and a `no_verdict`. `no_answer` is a classification that
|
||||
failed, so a `no_verdict` run with a `no_answer` check is the usual cause of
|
||||
a comment instead of a review. The `check` values are the names in a repo's
|
||||
rules file, at most 10 per repo.
|
||||
|
||||
The `review-classifier` dashboard in Grafana charts these.
|
||||
|
||||
## Agent-emitted custom metrics (`hive-metric`)
|
||||
|
||||
Agents can push arbitrary labeled metrics to the same OTEL collector via the
|
||||
|
|
|
|||
|
|
@ -674,6 +674,13 @@ the types of `claude_code.token.usage`, which `claude` exports on the claude
|
|||
runtimes, so the cache hit rate reads the same way on every runtime: `cacheRead`
|
||||
over the sum of `input`, `cacheRead` and `cacheCreation`.
|
||||
|
||||
The controller counts what requests and runs come to as the counters
|
||||
`hyperhive.review_classifier.requests`, `.verdicts` and `.checks` and the
|
||||
histogram `.run.duration`, and the service counts each ask it repeats because
|
||||
an answer failed validation as `hyperhive.review_classifier.ask.retries`. The
|
||||
[metrics table](../scheduler/observability.md#review-classifier-result-metrics-swarm-controller)
|
||||
lists their labels, and the `review-classifier` Grafana dashboard charts them.
|
||||
|
||||
The classifier logs each run's result to its journal, which ships to
|
||||
VictoriaLogs like every swarm service container's
|
||||
([`services.md`](services.md#logs-victorialogs)).
|
||||
|
|
|
|||
|
|
@ -207,14 +207,16 @@ kept.
|
|||
- `authelia`, `bao`, `queue`: the identity provider, the secret store and the NATS queue.
|
||||
- `claude-usage`: Claude use by model, effort, token type and source. Filter by hive, service (agents or the review classifier) and agent. The service filter separates the classifier's API-key spend from the agents.
|
||||
- `controller`: the swarm controller's job graph by state, and the push and commit rates it sees on the forge.
|
||||
- `review-classifier`: review requests by outcome, the verdict mix, results per check, retried asks, and run duration p50 and p95.
|
||||
- `forge`: what the forge holds, and the forge process's CPU, memory and descriptors.
|
||||
- `gateway`: the gateway nginx on each hive's host: requests per second, connections by state, and accepted against handled connections.
|
||||
- `collector`: what the OTEL collectors accept, refuse, export and queue, with queue depth against capacity.
|
||||
- `logstore`, `metricstore`: health of the log store and the metrics store.
|
||||
|
||||
A new panel can read "No data" until its source exists: the push and commit
|
||||
counters on `controller` appear after the first push webhook, and the refused
|
||||
and failure series on `collector` appear once something fails.
|
||||
counters on `controller` appear after the first push webhook, the refused
|
||||
and failure series on `collector` appear once something fails, and the
|
||||
`review-classifier` panels fill as review requests arrive.
|
||||
|
||||
**Where the data comes from.** The swarm's OTEL collector, below.
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue