Watch
0
0
Fork
You've already forked hyperhive
0

review classifier: run on ACP when the swarm settings choose it

The swarm settings gain `runtime` (`api-key`, the default, or `acp`) and
an `acp` provider record: an https base URL, the provider's model, and
its prices per million tokens. A PUT now changes only the fields its
body names, so the settings form that sends model, effort and limit
leaves the runtime alone; `null` clears `effort` or `acp`. The body stays
strict, and the stored document keeps #5192's lenient read.

The controller sends the runtime in each run request. On `acp` the
classifier runs one prompt on a new `opencode acp` session per
classification through hive-runtime's AcpRuntime, in a scratch
directory, with a config rendered from the settings. opencode runs
`read` and other tools without asking, so OPENCODE_PERMISSION
{"*":"deny"} takes every tool away from the model, and a deny-all policy
refuses any request on top. The title call, which resends the prompt,
is turned off. The deadline cancels the turn, and a turn that does not
end within 15s of that is dropped, which also covers an agent stuck
before its prompt. The answer is the final assistant text, as ACP has
no result event.

The provider key is read per run from
swarm/services/review-classifier/acp-provider, which the bao UI's
admins login may create and update, and reaches opencode through
AcpRuntime::with_provider_key and the trampoline's allowlist.

Each ACP classification's cost is computed from the configured prices
and the reported tokens, and added to
hyperhive.review_classifier.cost.usage labelled repo, pr, model and
currency.

Part of #5096
This commit is contained in:
atlas 2026-10-13 10:15:27 +02:00 • committed by mara
commit 1b4e764e5c
24 changed files with 1622 additions and 262 deletions

View file

@ -0,0 +1,150 @@
//! What a classification on the ACP runtime cost, exported as the counter
//! `hyperhive.review_classifier.cost.usage` with the run's `repo`, `pr`,
//! `model` and `currency`. On the claude runtime `claude` exports its own.
//!
//! The amount is computed here from the prices the run carries and the
//! tokens the agent reports, so it does not depend on how, or whether, the
//! agent prices a custom provider.
//!
//! The exporter is built on the first record, from the standard `OTEL_*`
//! environment, as `hive-agent`'s `otel_turn_metrics` does; with no
//! `OTEL_EXPORTER_OTLP_ENDPOINT` a record does nothing.
use std::sync::OnceLock;
use std::time::Duration;
use hive_runtime::TokenUsage;
use opentelemetry::KeyValue;
use opentelemetry::metrics::{Counter, MeterProvider as _};
use opentelemetry_otlp::{MetricExporter, Protocol, WithExportConfig};
use opentelemetry_sdk::metrics::{PeriodicReader, SdkMeterProvider};
use swarm_queue_client::review_classifier::Prices;
/// The currency every price is in.
pub const CURRENCY: &str = "USD";
/// How often the counter is exported.
const INTERVAL: Duration = Duration::from_mins(1);
/// One classification's cost, and what it is labelled with.
#[derive(Debug, Clone, PartialEq)]
pub struct Cost {
pub repo: String,
pub pr: u64,
pub model: String,
/// In [`CURRENCY`].
pub amount: f64,
}
/// Where a classification's cost goes: the metrics store in production, a
/// list in the tests.
pub trait CostSink: Send + Sync {
fn record(&self, cost: &Cost);
}
/// `usage` at `prices`. Cache reads and writes without a price of their own
/// are priced as input.
#[must_use]
pub fn amount(prices: &Prices, usage: &TokenUsage) -> f64 {
let per_token = |price: f64| price / 1_000_000.0;
// One model call's count is far below `u32::MAX`.
let tokens = |n: u64| f64::from(u32::try_from(n).unwrap_or(u32::MAX));
tokens(usage.input_tokens) * per_token(prices.input)
+ tokens(usage.output_tokens) * per_token(prices.output)
+ tokens(usage.cache_read_input_tokens)
* per_token(prices.cache_read.unwrap_or(prices.input))
+ tokens(usage.cache_creation_input_tokens)
* per_token(prices.cache_write.unwrap_or(prices.input))
}
/// The counter in the metrics store.
pub struct Otel;
impl CostSink for Otel {
fn record(&self, cost: &Cost) {
static COUNTER: OnceLock<Option<(SdkMeterProvider, Counter<f64>)>> = OnceLock::new();
let Some((_, counter)) = COUNTER.get_or_init(build) else {
return;
};
counter.add(
cost.amount,
&[
KeyValue::new("repo", cost.repo.clone()),
KeyValue::new("pr", i64::try_from(cost.pr).unwrap_or(i64::MAX)),
KeyValue::new("model", cost.model.clone()),
KeyValue::new("currency", CURRENCY),
],
);
}
}
/// The provider, kept alive with the counter since it exports only while it
/// lives, or `None` when no collector is configured or the exporter failed.
fn build() -> Option<(SdkMeterProvider, Counter<f64>)> {
if !std::env::var("OTEL_EXPORTER_OTLP_ENDPOINT").is_ok_and(|s| !s.trim().is_empty()) {
return None;
}
// The blocking client: `PeriodicReader` exports from a thread of its own
// with no Tokio reactor. The resource comes from `OTEL_RESOURCE_ATTRIBUTES`.
let exporter = MetricExporter::builder()
.with_http()
.with_protocol(Protocol::HttpJson)
.build();
let exporter = match exporter {
Ok(exporter) => exporter,
Err(e) => {
tracing::warn!(error = %e, "building the cost metric exporter failed; no cost is exported");
return None;
}
};
let reader = PeriodicReader::builder(exporter)
.with_interval(INTERVAL)
.build();
let provider = SdkMeterProvider::builder().with_reader(reader).build();
let counter = provider
.meter("hyperhive.review_classifier")
.f64_counter("hyperhive.review_classifier.cost.usage")
.build();
Some((provider, counter))
}
#[cfg(test)]
mod tests {
use super::*;
fn usage(input: u64, output: u64, cache_read: u64, cache_write: u64) -> TokenUsage {
let mut usage = TokenUsage::default();
usage.input_tokens = input;
usage.output_tokens = output;
usage.cache_read_input_tokens = cache_read;
usage.cache_creation_input_tokens = cache_write;
usage
}
#[test]
fn each_token_class_is_charged_at_its_price_per_million() {
let prices = Prices {
input: 3.0,
output: 15.0,
cache_read: Some(0.3),
cache_write: Some(3.75),
};
let cost = amount(&prices, &usage(1_000, 200, 10_000, 2_000));
assert!(
(cost - (0.003 + 0.003 + 0.003 + 0.0075)).abs() < 1e-12,
"{cost}"
);
}
#[test]
fn cache_without_a_price_of_its_own_is_charged_as_input() {
let prices = Prices {
input: 2.0,
output: 0.0,
cache_read: None,
cache_write: None,
};
let cost = amount(&prices, &usage(0, 0, 500_000, 500_000));
assert!((cost - 2.0).abs() < 1e-12, "{cost}");
}
}