review classifier: run on ACP when the swarm settings choose it
The swarm settings gain `runtime` (`api-key`, the default, or `acp`) and an `acp` provider record: an https base URL, the provider's model, and its prices per million tokens. A PUT now changes only the fields its body names, so the settings form that sends model, effort and limit leaves the runtime alone; `null` clears `effort` or `acp`. The body stays strict, and the stored document keeps #5192's lenient read. The controller sends the runtime in each run request. On `acp` the classifier runs one prompt on a new `opencode acp` session per classification through hive-runtime's AcpRuntime, in a scratch directory, with a config rendered from the settings. opencode runs `read` and other tools without asking, so OPENCODE_PERMISSION {"*":"deny"} takes every tool away from the model, and a deny-all policy refuses any request on top. The title call, which resends the prompt, is turned off. The deadline cancels the turn, and a turn that does not end within 15s of that is dropped, which also covers an agent stuck before its prompt. The answer is the final assistant text, as ACP has no result event. The provider key is read per run from swarm/services/review-classifier/acp-provider, which the bao UI's admins login may create and update, and reaches opencode through AcpRuntime::with_provider_key and the trampoline's allowlist. Each ACP classification's cost is computed from the configured prices and the reported tokens, and added to hyperhive.review_classifier.cost.usage labelled repo, pr, model and currency. Part of #5096
This commit is contained in:
parent
38ee68c31d
commit
1b4e764e5c
24 changed files with 1622 additions and 262 deletions
150
swarm-review-classifier/src/cost.rs
Normal file
150
swarm-review-classifier/src/cost.rs
Normal file
|
|
@ -0,0 +1,150 @@
|
|||
//! What a classification on the ACP runtime cost, exported as the counter
|
||||
//! `hyperhive.review_classifier.cost.usage` with the run's `repo`, `pr`,
|
||||
//! `model` and `currency`. On the claude runtime `claude` exports its own.
|
||||
//!
|
||||
//! The amount is computed here from the prices the run carries and the
|
||||
//! tokens the agent reports, so it does not depend on how, or whether, the
|
||||
//! agent prices a custom provider.
|
||||
//!
|
||||
//! The exporter is built on the first record, from the standard `OTEL_*`
|
||||
//! environment, as `hive-agent`'s `otel_turn_metrics` does; with no
|
||||
//! `OTEL_EXPORTER_OTLP_ENDPOINT` a record does nothing.
|
||||
|
||||
use std::sync::OnceLock;
|
||||
use std::time::Duration;
|
||||
|
||||
use hive_runtime::TokenUsage;
|
||||
use opentelemetry::KeyValue;
|
||||
use opentelemetry::metrics::{Counter, MeterProvider as _};
|
||||
use opentelemetry_otlp::{MetricExporter, Protocol, WithExportConfig};
|
||||
use opentelemetry_sdk::metrics::{PeriodicReader, SdkMeterProvider};
|
||||
use swarm_queue_client::review_classifier::Prices;
|
||||
|
||||
/// The currency every price is in.
|
||||
pub const CURRENCY: &str = "USD";
|
||||
|
||||
/// How often the counter is exported.
|
||||
const INTERVAL: Duration = Duration::from_mins(1);
|
||||
|
||||
/// One classification's cost, and what it is labelled with.
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub struct Cost {
|
||||
pub repo: String,
|
||||
pub pr: u64,
|
||||
pub model: String,
|
||||
/// In [`CURRENCY`].
|
||||
pub amount: f64,
|
||||
}
|
||||
|
||||
/// Where a classification's cost goes: the metrics store in production, a
|
||||
/// list in the tests.
|
||||
pub trait CostSink: Send + Sync {
|
||||
fn record(&self, cost: &Cost);
|
||||
}
|
||||
|
||||
/// `usage` at `prices`. Cache reads and writes without a price of their own
|
||||
/// are priced as input.
|
||||
#[must_use]
|
||||
pub fn amount(prices: &Prices, usage: &TokenUsage) -> f64 {
|
||||
let per_token = |price: f64| price / 1_000_000.0;
|
||||
// One model call's count is far below `u32::MAX`.
|
||||
let tokens = |n: u64| f64::from(u32::try_from(n).unwrap_or(u32::MAX));
|
||||
tokens(usage.input_tokens) * per_token(prices.input)
|
||||
+ tokens(usage.output_tokens) * per_token(prices.output)
|
||||
+ tokens(usage.cache_read_input_tokens)
|
||||
* per_token(prices.cache_read.unwrap_or(prices.input))
|
||||
+ tokens(usage.cache_creation_input_tokens)
|
||||
* per_token(prices.cache_write.unwrap_or(prices.input))
|
||||
}
|
||||
|
||||
/// The counter in the metrics store.
|
||||
pub struct Otel;
|
||||
|
||||
impl CostSink for Otel {
|
||||
fn record(&self, cost: &Cost) {
|
||||
static COUNTER: OnceLock<Option<(SdkMeterProvider, Counter<f64>)>> = OnceLock::new();
|
||||
let Some((_, counter)) = COUNTER.get_or_init(build) else {
|
||||
return;
|
||||
};
|
||||
counter.add(
|
||||
cost.amount,
|
||||
&[
|
||||
KeyValue::new("repo", cost.repo.clone()),
|
||||
KeyValue::new("pr", i64::try_from(cost.pr).unwrap_or(i64::MAX)),
|
||||
KeyValue::new("model", cost.model.clone()),
|
||||
KeyValue::new("currency", CURRENCY),
|
||||
],
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// The provider, kept alive with the counter since it exports only while it
|
||||
/// lives, or `None` when no collector is configured or the exporter failed.
|
||||
fn build() -> Option<(SdkMeterProvider, Counter<f64>)> {
|
||||
if !std::env::var("OTEL_EXPORTER_OTLP_ENDPOINT").is_ok_and(|s| !s.trim().is_empty()) {
|
||||
return None;
|
||||
}
|
||||
// The blocking client: `PeriodicReader` exports from a thread of its own
|
||||
// with no Tokio reactor. The resource comes from `OTEL_RESOURCE_ATTRIBUTES`.
|
||||
let exporter = MetricExporter::builder()
|
||||
.with_http()
|
||||
.with_protocol(Protocol::HttpJson)
|
||||
.build();
|
||||
let exporter = match exporter {
|
||||
Ok(exporter) => exporter,
|
||||
Err(e) => {
|
||||
tracing::warn!(error = %e, "building the cost metric exporter failed; no cost is exported");
|
||||
return None;
|
||||
}
|
||||
};
|
||||
let reader = PeriodicReader::builder(exporter)
|
||||
.with_interval(INTERVAL)
|
||||
.build();
|
||||
let provider = SdkMeterProvider::builder().with_reader(reader).build();
|
||||
let counter = provider
|
||||
.meter("hyperhive.review_classifier")
|
||||
.f64_counter("hyperhive.review_classifier.cost.usage")
|
||||
.build();
|
||||
Some((provider, counter))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn usage(input: u64, output: u64, cache_read: u64, cache_write: u64) -> TokenUsage {
|
||||
let mut usage = TokenUsage::default();
|
||||
usage.input_tokens = input;
|
||||
usage.output_tokens = output;
|
||||
usage.cache_read_input_tokens = cache_read;
|
||||
usage.cache_creation_input_tokens = cache_write;
|
||||
usage
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn each_token_class_is_charged_at_its_price_per_million() {
|
||||
let prices = Prices {
|
||||
input: 3.0,
|
||||
output: 15.0,
|
||||
cache_read: Some(0.3),
|
||||
cache_write: Some(3.75),
|
||||
};
|
||||
let cost = amount(&prices, &usage(1_000, 200, 10_000, 2_000));
|
||||
assert!(
|
||||
(cost - (0.003 + 0.003 + 0.003 + 0.0075)).abs() < 1e-12,
|
||||
"{cost}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cache_without_a_price_of_its_own_is_charged_as_input() {
|
||||
let prices = Prices {
|
||||
input: 2.0,
|
||||
output: 0.0,
|
||||
cache_read: None,
|
||||
cache_write: None,
|
||||
};
|
||||
let cost = amount(&prices, &usage(0, 0, 500_000, 500_000));
|
||||
assert!((cost - 2.0).abs() < 1e-12, "{cost}");
|
||||
}
|
||||
}
|
||||
Loading…
Reference in a new issue