stats: per-agent /stats page with chart.js trends + breakdowns
new hive-ag3nt::stats module reads turn_stats.sqlite read-only and aggregates over 24h/7d/30d windows (hourly/daily buckets) — turn rate, p50/p95/avg duration, ctx tokens (avg/max), cost token components, top tools, wake mix, result mix. served by the agent itself so per-MCP extensions can register more providers without the host knowing their schemas. /stats route + /api/stats?window=... on the per-agent web ui. chart.js v4.4.4 pulled from jsdelivr (SRI hash deferred). nav links: 📊 chip on the dashboard container row + 📊 stats → on the per-agent header. todo housekeeping: softened damocles-area note at the top, new reverse-proxy + deferred reminder-rollup items, removed the two telemetry-ui items absorbed by this page.
This commit is contained in:
müde
parent
f9f1346eae
commit
d3f90f4cc0
8 changed files with 930 additions and 3 deletions
10
TODO.md
10
TODO.md
|
|
@ -1,5 +1,10 @@
|
|||
# Hyperhive TODOs
|
||||
|
||||
> **Rough split for who picks up what:** harness ergonomics + host-side
|
||||
> harness plumbing tend to be damocles' interest area; UI-polish work
|
||||
> for the operator is not. Use that as a hint when picking up items,
|
||||
> not a hard rule.
|
||||
|
||||
## Architecture / Features
|
||||
|
||||
- Shared space for all agents to access documents/files without manager routing
|
||||
|
|
@ -18,7 +23,8 @@
|
|||
|
||||
## Dashboard
|
||||
|
||||
- **Delivered-reminder rollups**: per-agent delivered-count chip (last 24h) + histogram of attempts-vs-successes on the container row. Needs `Broker::count_delivered_reminders_since(agent, ts)` (cheap COUNT against the `reminders` table, `WHERE agent = ?1 AND sent_at >= ?2`).
|
||||
- **Unified URL scheme via reverse proxy**: today every agent's web UI is reached at `<host>:<per-agent-port>/`, so operators juggle a port list. Stand up nginx (or similar) terminating one domain that fans requests to `/agent/<name>/...` out to each container's web port, and to `/` for the main dashboard. Touches: a NixOS module on the host, the dashboard's per-agent link rendering, and the per-agent web server's base-path handling (currently assumes root). Lets bookmarks survive port reshuffles and unblocks per-agent stats links being relative URLs instead of hard-coded ports.
|
||||
- **Delivered-reminder rollup on the per-agent stats page**: surface attempt / success / failure counts for reminders this agent fired (in the existing `/stats` page). Needs an `AgentRequest::ReminderRollup { since_secs }` / matching `ManagerRequest::ReminderRollup` RPC so the agent can pull the counts from the host's broker DB (the reminders table is host-owned; agent state doesn't have them). Deferred from the initial stats page so the first cut stays self-contained to data the agent already owns.
|
||||
|
||||
## Security
|
||||
|
||||
|
|
@ -49,8 +55,6 @@ how often the friction bites in normal use.
|
|||
## Telemetry
|
||||
|
||||
- **Per-turn stats: host-side vacuum sweep**: the sink writes to `/state/hyperhive-turn-stats.sqlite` on each agent's state dir; needs a periodic retention sweep mirroring `events_vacuum.rs` so the table doesn't grow forever. Default keep-window: 90 days (turn-stats are denser than events but smaller per-row, ~200B each).
|
||||
- **Surface per-turn stats on the agent web UI**: "N turns today" chip + rolling tool-call histogram tooltip on the model chip. (`open_threads` and `open_reminders` chips already landed via other paths — open-threads section on the page + reminder count chip on the container row.) Reads the per-agent `turn_stats.sqlite`.
|
||||
- **Stats UI on the main dashboard**: per-agent rollups (avg turn duration, tokens-since-boot, top 5 tools) on the container row. Same data source, host-side aggregation query.
|
||||
|
||||
## Harness Behaviour
|
||||
|
||||
|
|
|
|||
|
|
@ -8,6 +8,7 @@
|
|||
<body>
|
||||
<pre class="banner" id="banner">░▒▓█▓▒░ … ░▒▓█▓▒░ hyperhive ag3nt ░▒▓█▓▒░</pre>
|
||||
<h2 id="title">◆ … ◆</h2>
|
||||
<p class="meta"><a href="/stats" style="color: var(--cyan); text-decoration: none;">📊 stats →</a></p>
|
||||
|
||||
<div id="status">
|
||||
<p class="meta">loading…</p>
|
||||
|
|
|
|||
76
hive-ag3nt/assets/stats.html
Normal file
76
hive-ag3nt/assets/stats.html
Normal file
|
|
@ -0,0 +1,76 @@
|
|||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<title>hyperhive agent — stats</title>
|
||||
<link rel="stylesheet" href="/static/agent.css">
|
||||
<style>
|
||||
.stats-nav { display: flex; gap: 0.75rem; align-items: baseline; margin-bottom: 0.5rem; }
|
||||
.stats-nav a { color: var(--cyan); text-decoration: none; }
|
||||
.stats-nav a:hover { text-decoration: underline; }
|
||||
.window-tabs { display: flex; gap: 0.4rem; margin: 0.5rem 0 1rem; }
|
||||
.window-tabs button {
|
||||
background: var(--bg-elev); color: var(--fg);
|
||||
border: 1px solid var(--border); padding: 0.3rem 0.8rem;
|
||||
font-family: inherit; cursor: pointer;
|
||||
}
|
||||
.window-tabs button.active { background: var(--purple-dim); border-color: var(--purple); color: var(--purple); }
|
||||
.summary { display: flex; gap: 0.75rem; flex-wrap: wrap; margin-bottom: 1rem; }
|
||||
.summary .chip {
|
||||
background: var(--bg-elev); border: 1px solid var(--border);
|
||||
padding: 0.4rem 0.8rem; border-radius: 4px;
|
||||
}
|
||||
.summary .chip .label { color: var(--muted); margin-right: 0.5rem; }
|
||||
.summary .chip .value { color: var(--cyan); font-weight: bold; }
|
||||
.grid {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(auto-fit, minmax(420px, 1fr));
|
||||
gap: 1rem;
|
||||
}
|
||||
.card {
|
||||
background: var(--bg-elev);
|
||||
border: 1px solid var(--border);
|
||||
padding: 0.75rem 1rem 1rem;
|
||||
border-radius: 4px;
|
||||
}
|
||||
.card h3 { margin: 0 0 0.5rem; color: var(--purple); font-size: 0.95rem; font-weight: normal; }
|
||||
.card .chart-wrap { position: relative; height: 220px; }
|
||||
.card.wide { grid-column: 1 / -1; }
|
||||
.card.wide .chart-wrap { height: 260px; }
|
||||
.empty-note { color: var(--muted); font-style: italic; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<pre class="banner">░▒▓█▓▒░ … ░▒▓█▓▒░ hyperhive ag3nt · stats ░▒▓█▓▒░</pre>
|
||||
<div class="stats-nav">
|
||||
<a id="back-link" href="/">← live</a>
|
||||
<a id="dashboard-link" href="#">dashboard</a>
|
||||
<h2 id="title" style="margin: 0;">◆ … ◆</h2>
|
||||
</div>
|
||||
|
||||
<div class="window-tabs" id="window-tabs">
|
||||
<button data-w="24h" class="active">last 24h</button>
|
||||
<button data-w="7d">last 7d</button>
|
||||
<button data-w="30d">last 30d</button>
|
||||
</div>
|
||||
|
||||
<div class="summary" id="summary"></div>
|
||||
|
||||
<div class="grid">
|
||||
<div class="card wide"><h3>turns per bucket</h3><div class="chart-wrap"><canvas id="chart-turns"></canvas></div></div>
|
||||
<div class="card wide"><h3>turn duration (ms) — p50 / p95 / avg</h3><div class="chart-wrap"><canvas id="chart-duration"></canvas></div></div>
|
||||
<div class="card wide"><h3>context tokens (last inference per turn) — avg / max</h3><div class="chart-wrap"><canvas id="chart-ctx"></canvas></div></div>
|
||||
<div class="card wide"><h3>token cost per bucket (sum across inferences)</h3><div class="chart-wrap"><canvas id="chart-cost"></canvas></div></div>
|
||||
<div class="card"><h3>top tools</h3><div class="chart-wrap"><canvas id="chart-tools"></canvas></div></div>
|
||||
<div class="card"><h3>wake source mix</h3><div class="chart-wrap"><canvas id="chart-wake"></canvas></div></div>
|
||||
<div class="card"><h3>result mix</h3><div class="chart-wrap"><canvas id="chart-result"></canvas></div></div>
|
||||
</div>
|
||||
|
||||
<!-- Chart.js pinned to a fixed version from jsDelivr. SRI hash is
|
||||
not set yet — add an integrity="sha384-..." attribute when we
|
||||
have a way to compute it deterministically in the build. -->
|
||||
<script src="https://cdn.jsdelivr.net/npm/chart.js@4.4.4/dist/chart.umd.min.js"
|
||||
crossorigin="anonymous"></script>
|
||||
<script src="/static/stats.js" defer></script>
|
||||
</body>
|
||||
</html>
|
||||
308
hive-ag3nt/assets/stats.js
Normal file
308
hive-ag3nt/assets/stats.js
Normal file
|
|
@ -0,0 +1,308 @@
|
|||
// Per-agent stats page. Fetches /api/state for the title + dashboard link
|
||||
// once on load, then /api/stats?window=... for the chart data — re-fetches
|
||||
// when the operator clicks a window tab.
|
||||
|
||||
(function () {
|
||||
'use strict';
|
||||
|
||||
const cssVar = (name) => getComputedStyle(document.documentElement).getPropertyValue(name).trim();
|
||||
const palette = {
|
||||
bg: cssVar('--bg'),
|
||||
bgElev: cssVar('--bg-elev'),
|
||||
fg: cssVar('--fg'),
|
||||
muted: cssVar('--muted'),
|
||||
purple: cssVar('--purple'),
|
||||
cyan: cssVar('--cyan'),
|
||||
pink: cssVar('--pink'),
|
||||
amber: cssVar('--amber'),
|
||||
green: cssVar('--green'),
|
||||
red: cssVar('--red'),
|
||||
border: cssVar('--border'),
|
||||
};
|
||||
// Distinct hues for categorical charts (top tools / wake mix / result mix).
|
||||
const wheel = [palette.purple, palette.cyan, palette.pink, palette.amber,
|
||||
palette.green, palette.red, '#94e2d5', '#f9e2af',
|
||||
'#74c7ec', '#b4befe'];
|
||||
|
||||
// Apply Catppuccin defaults globally so each Chart inherits without per-call
|
||||
// overrides. Chart.js v4 reads these on chart construction.
|
||||
Chart.defaults.color = palette.fg;
|
||||
Chart.defaults.borderColor = palette.border;
|
||||
Chart.defaults.font.family = '"JetBrains Mono", "Fira Code", monospace';
|
||||
Chart.defaults.font.size = 11;
|
||||
Chart.defaults.plugins.legend.labels.color = palette.fg;
|
||||
|
||||
const charts = {};
|
||||
let currentWindow = '24h';
|
||||
|
||||
function fmtMs(ms) {
|
||||
if (!Number.isFinite(ms) || ms <= 0) return '0';
|
||||
if (ms < 1000) return ms.toFixed(0) + 'ms';
|
||||
return (ms / 1000).toFixed(ms < 10000 ? 2 : 1) + 's';
|
||||
}
|
||||
|
||||
function fmtInt(n) {
|
||||
if (!Number.isFinite(n)) return '0';
|
||||
return new Intl.NumberFormat().format(Math.round(n));
|
||||
}
|
||||
|
||||
function bucketLabel(ts, bucketSecs) {
|
||||
const d = new Date(ts * 1000);
|
||||
if (bucketSecs >= 86400) {
|
||||
return d.toISOString().slice(5, 10); // MM-DD
|
||||
}
|
||||
return d.toISOString().slice(11, 16); // HH:MM
|
||||
}
|
||||
|
||||
function destroy(name) {
|
||||
if (charts[name]) {
|
||||
charts[name].destroy();
|
||||
delete charts[name];
|
||||
}
|
||||
}
|
||||
|
||||
function paintEmpty(canvasId, msg) {
|
||||
destroy(canvasId);
|
||||
const cv = document.getElementById(canvasId);
|
||||
if (!cv) return;
|
||||
const ctx = cv.getContext('2d');
|
||||
ctx.clearRect(0, 0, cv.width, cv.height);
|
||||
ctx.fillStyle = palette.muted;
|
||||
ctx.font = '12px monospace';
|
||||
ctx.textAlign = 'center';
|
||||
ctx.textBaseline = 'middle';
|
||||
ctx.fillText(msg, cv.width / 2, cv.height / 2);
|
||||
}
|
||||
|
||||
function renderSummary(s) {
|
||||
const root = document.getElementById('summary');
|
||||
root.replaceChildren();
|
||||
const chips = [
|
||||
['turns', fmtInt(s.turn_count)],
|
||||
['avg duration', fmtMs(s.duration_summary.avg_ms)],
|
||||
['p50 duration', fmtMs(s.duration_summary.p50_ms)],
|
||||
['p95 duration', fmtMs(s.duration_summary.p95_ms)],
|
||||
['window', s.window],
|
||||
];
|
||||
for (const [label, value] of chips) {
|
||||
const chip = document.createElement('span');
|
||||
chip.className = 'chip';
|
||||
const l = document.createElement('span');
|
||||
l.className = 'label';
|
||||
l.textContent = label;
|
||||
const v = document.createElement('span');
|
||||
v.className = 'value';
|
||||
v.textContent = value;
|
||||
chip.append(l, v);
|
||||
root.append(chip);
|
||||
}
|
||||
}
|
||||
|
||||
function renderTurnsChart(s) {
|
||||
const id = 'chart-turns';
|
||||
destroy(id);
|
||||
const labels = s.buckets.map((b) => bucketLabel(b.ts, s.bucket_seconds));
|
||||
const data = s.buckets.map((b) => b.turn_count);
|
||||
charts[id] = new Chart(document.getElementById(id), {
|
||||
type: 'bar',
|
||||
data: {
|
||||
labels,
|
||||
datasets: [{
|
||||
label: 'turns',
|
||||
data,
|
||||
backgroundColor: palette.purple,
|
||||
borderColor: palette.purple,
|
||||
borderWidth: 1,
|
||||
}],
|
||||
},
|
||||
options: {
|
||||
responsive: true, maintainAspectRatio: false,
|
||||
plugins: { legend: { display: false } },
|
||||
scales: {
|
||||
x: { grid: { color: palette.border } },
|
||||
y: { beginAtZero: true, grid: { color: palette.border }, ticks: { precision: 0 } },
|
||||
},
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
function renderDurationChart(s) {
|
||||
const id = 'chart-duration';
|
||||
destroy(id);
|
||||
const labels = s.buckets.map((b) => bucketLabel(b.ts, s.bucket_seconds));
|
||||
const ds = (label, color, key) => ({
|
||||
label, data: s.buckets.map((b) => b[key]),
|
||||
borderColor: color, backgroundColor: color + '33',
|
||||
tension: 0.25, pointRadius: 0, borderWidth: 2, spanGaps: true,
|
||||
});
|
||||
charts[id] = new Chart(document.getElementById(id), {
|
||||
type: 'line',
|
||||
data: {
|
||||
labels,
|
||||
datasets: [
|
||||
ds('p50', palette.cyan, 'p50_duration_ms'),
|
||||
ds('p95', palette.pink, 'p95_duration_ms'),
|
||||
ds('avg', palette.amber, 'avg_duration_ms'),
|
||||
],
|
||||
},
|
||||
options: {
|
||||
responsive: true, maintainAspectRatio: false,
|
||||
scales: {
|
||||
x: { grid: { color: palette.border } },
|
||||
y: {
|
||||
beginAtZero: true,
|
||||
grid: { color: palette.border },
|
||||
ticks: { callback: (v) => fmtMs(v) },
|
||||
},
|
||||
},
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
function renderCtxChart(s) {
|
||||
const id = 'chart-ctx';
|
||||
destroy(id);
|
||||
const labels = s.buckets.map((b) => bucketLabel(b.ts, s.bucket_seconds));
|
||||
charts[id] = new Chart(document.getElementById(id), {
|
||||
type: 'line',
|
||||
data: {
|
||||
labels,
|
||||
datasets: [
|
||||
{
|
||||
label: 'avg ctx',
|
||||
data: s.buckets.map((b) => b.avg_ctx_tokens),
|
||||
borderColor: palette.cyan,
|
||||
backgroundColor: palette.cyan + '33',
|
||||
tension: 0.25, pointRadius: 0, borderWidth: 2, spanGaps: true,
|
||||
},
|
||||
{
|
||||
label: 'max ctx',
|
||||
data: s.buckets.map((b) => b.max_ctx_tokens),
|
||||
borderColor: palette.amber,
|
||||
backgroundColor: palette.amber + '33',
|
||||
tension: 0.25, pointRadius: 0, borderWidth: 2, spanGaps: true,
|
||||
},
|
||||
],
|
||||
},
|
||||
options: {
|
||||
responsive: true, maintainAspectRatio: false,
|
||||
scales: {
|
||||
x: { grid: { color: palette.border } },
|
||||
y: { beginAtZero: true, grid: { color: palette.border }, ticks: { callback: (v) => fmtInt(v) } },
|
||||
},
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
function renderCostChart(s) {
|
||||
const id = 'chart-cost';
|
||||
destroy(id);
|
||||
const labels = s.buckets.map((b) => bucketLabel(b.ts, s.bucket_seconds));
|
||||
// Stacked bars: cache_read (cheap) / cache_creation / input / output.
|
||||
// Highlights "what's actually getting billed at full rate" vs cache hits.
|
||||
charts[id] = new Chart(document.getElementById(id), {
|
||||
type: 'bar',
|
||||
data: {
|
||||
labels,
|
||||
datasets: [
|
||||
{ label: 'cache_read', data: s.buckets.map((b) => b.cache_read_input_tokens),
|
||||
backgroundColor: palette.muted },
|
||||
{ label: 'cache_creation', data: s.buckets.map((b) => b.cache_creation_input_tokens),
|
||||
backgroundColor: palette.cyan },
|
||||
{ label: 'input', data: s.buckets.map((b) => b.input_tokens),
|
||||
backgroundColor: palette.amber },
|
||||
{ label: 'output', data: s.buckets.map((b) => b.output_tokens),
|
||||
backgroundColor: palette.pink },
|
||||
],
|
||||
},
|
||||
options: {
|
||||
responsive: true, maintainAspectRatio: false,
|
||||
scales: {
|
||||
x: { stacked: true, grid: { color: palette.border } },
|
||||
y: { stacked: true, beginAtZero: true,
|
||||
grid: { color: palette.border }, ticks: { callback: (v) => fmtInt(v) } },
|
||||
},
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
function renderKeyCount(canvasId, items, emptyMsg) {
|
||||
destroy(canvasId);
|
||||
if (!items || items.length === 0) {
|
||||
paintEmpty(canvasId, emptyMsg);
|
||||
return;
|
||||
}
|
||||
const labels = items.map((kc) => kc.key);
|
||||
const data = items.map((kc) => kc.count);
|
||||
const colors = items.map((_, i) => wheel[i % wheel.length]);
|
||||
charts[canvasId] = new Chart(document.getElementById(canvasId), {
|
||||
type: 'doughnut',
|
||||
data: { labels, datasets: [{ data, backgroundColor: colors, borderColor: palette.bg, borderWidth: 2 }] },
|
||||
options: {
|
||||
responsive: true, maintainAspectRatio: false,
|
||||
plugins: { legend: { position: 'right', labels: { boxWidth: 12 } } },
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
function render(s) {
|
||||
renderSummary(s);
|
||||
if (s.turn_count === 0) {
|
||||
paintEmpty('chart-turns', 'no turns in window');
|
||||
paintEmpty('chart-duration', 'no turns in window');
|
||||
paintEmpty('chart-ctx', 'no turns in window');
|
||||
paintEmpty('chart-cost', 'no turns in window');
|
||||
paintEmpty('chart-tools', 'no tool calls');
|
||||
paintEmpty('chart-wake', 'no wakes');
|
||||
paintEmpty('chart-result', 'no results');
|
||||
return;
|
||||
}
|
||||
renderTurnsChart(s);
|
||||
renderDurationChart(s);
|
||||
renderCtxChart(s);
|
||||
renderCostChart(s);
|
||||
renderKeyCount('chart-tools', s.tool_breakdown, 'no tool calls');
|
||||
renderKeyCount('chart-wake', s.wake_mix, 'no wakes');
|
||||
renderKeyCount('chart-result', s.result_mix, 'no results');
|
||||
}
|
||||
|
||||
async function loadStats() {
|
||||
try {
|
||||
const resp = await fetch('/api/stats?window=' + encodeURIComponent(currentWindow));
|
||||
if (!resp.ok) throw new Error('http ' + resp.status);
|
||||
const snap = await resp.json();
|
||||
render(snap);
|
||||
} catch (e) {
|
||||
document.getElementById('summary').textContent = 'stats fetch failed: ' + e;
|
||||
}
|
||||
}
|
||||
|
||||
async function loadIdentity() {
|
||||
try {
|
||||
const resp = await fetch('/api/state');
|
||||
if (!resp.ok) return;
|
||||
const s = await resp.json();
|
||||
document.title = 'stats · ' + s.label;
|
||||
document.getElementById('title').textContent = '◆ ' + s.label + ' ◆';
|
||||
const dl = document.getElementById('dashboard-link');
|
||||
dl.href = 'http://' + window.location.hostname + ':' + s.dashboard_port + '/';
|
||||
} catch (_) { /* non-fatal */ }
|
||||
}
|
||||
|
||||
function bindTabs() {
|
||||
const tabs = document.getElementById('window-tabs');
|
||||
tabs.addEventListener('click', (ev) => {
|
||||
const btn = ev.target.closest('button[data-w]');
|
||||
if (!btn) return;
|
||||
currentWindow = btn.dataset.w;
|
||||
for (const b of tabs.querySelectorAll('button')) b.classList.toggle('active', b === btn);
|
||||
loadStats();
|
||||
});
|
||||
}
|
||||
|
||||
document.addEventListener('DOMContentLoaded', () => {
|
||||
bindTabs();
|
||||
loadIdentity();
|
||||
loadStats();
|
||||
});
|
||||
})();
|
||||
|
|
@ -8,6 +8,7 @@ pub mod login_session;
|
|||
pub mod mcp;
|
||||
pub mod paths;
|
||||
pub mod plugins;
|
||||
pub mod stats;
|
||||
pub mod turn;
|
||||
pub mod turn_stats;
|
||||
pub mod web_ui;
|
||||
|
|
|
|||
500
hive-ag3nt/src/stats.rs
Normal file
500
hive-ag3nt/src/stats.rs
Normal file
|
|
@ -0,0 +1,500 @@
|
|||
//! Read-side aggregations over the per-agent `turn_stats.sqlite` for
|
||||
//! the agent's `/stats` web page. Owned by the agent (same process
|
||||
//! that writes the sink) so per-MCP extensions can register more
|
||||
//! providers without the host needing to know their schemas.
|
||||
//!
|
||||
//! Best-effort: any sqlite error returns an empty snapshot rather than
|
||||
//! propagating — the stats page is decorative, not authoritative, and
|
||||
//! a missing db on a brand-new agent shouldn't 500 the route.
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use anyhow::{Context, Result};
|
||||
use rusqlite::{Connection, OpenFlags};
|
||||
use serde::Serialize;
|
||||
|
||||
/// Window param accepted by `/api/stats?window=`. Each maps to a
|
||||
/// total span + the bucket width used to roll up trend series.
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
pub enum Window {
|
||||
Day,
|
||||
Week,
|
||||
Month,
|
||||
}
|
||||
|
||||
impl Window {
|
||||
pub fn parse(s: &str) -> Self {
|
||||
match s {
|
||||
"7d" => Self::Week,
|
||||
"30d" => Self::Month,
|
||||
_ => Self::Day,
|
||||
}
|
||||
}
|
||||
|
||||
fn label(self) -> &'static str {
|
||||
match self {
|
||||
Self::Day => "24h",
|
||||
Self::Week => "7d",
|
||||
Self::Month => "30d",
|
||||
}
|
||||
}
|
||||
|
||||
fn span_secs(self) -> i64 {
|
||||
match self {
|
||||
Self::Day => 24 * 3600,
|
||||
Self::Week => 7 * 24 * 3600,
|
||||
Self::Month => 30 * 24 * 3600,
|
||||
}
|
||||
}
|
||||
|
||||
fn bucket_secs(self) -> i64 {
|
||||
match self {
|
||||
// hourly for the 24h view, daily for the longer ranges.
|
||||
Self::Day => 3600,
|
||||
Self::Week | Self::Month => 24 * 3600,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Serialize)]
|
||||
pub struct Snapshot {
|
||||
pub window: &'static str,
|
||||
pub bucket_seconds: i64,
|
||||
pub now: i64,
|
||||
pub from: i64,
|
||||
/// Total turns in the window.
|
||||
pub turn_count: u64,
|
||||
/// Time-bucketed trend series, oldest first. Always covers the
|
||||
/// full window even for empty buckets (so charts paint a stable
|
||||
/// x-axis instead of skipping gaps).
|
||||
pub buckets: Vec<Bucket>,
|
||||
/// Top tools by call count across the window. Capped to 10.
|
||||
pub tool_breakdown: Vec<KeyCount>,
|
||||
pub wake_mix: Vec<KeyCount>,
|
||||
pub result_mix: Vec<KeyCount>,
|
||||
/// Across-window p50 / p95 / avg of `duration_ms`. Same numbers
|
||||
/// as the per-bucket fields but aggregated over the whole window
|
||||
/// for the headline summary chips.
|
||||
pub duration_summary: DurationSummary,
|
||||
}
|
||||
|
||||
#[derive(Debug, Serialize)]
|
||||
pub struct Bucket {
|
||||
/// Unix timestamp of the bucket start.
|
||||
pub ts: i64,
|
||||
pub turn_count: u64,
|
||||
pub avg_duration_ms: f64,
|
||||
pub p50_duration_ms: f64,
|
||||
pub p95_duration_ms: f64,
|
||||
/// Sums across the bucket. JS picks how to combine them
|
||||
/// (input + output for cost, etc.) so we don't bake a policy in.
|
||||
pub input_tokens: u64,
|
||||
pub output_tokens: u64,
|
||||
pub cache_read_input_tokens: u64,
|
||||
pub cache_creation_input_tokens: u64,
|
||||
/// Mean of `last_input_tokens` across the bucket (the context
|
||||
/// size at turn-end — useful for spotting drift toward compaction).
|
||||
pub avg_ctx_tokens: f64,
|
||||
pub max_ctx_tokens: u64,
|
||||
}
|
||||
|
||||
#[derive(Debug, Serialize)]
|
||||
pub struct KeyCount {
|
||||
pub key: String,
|
||||
pub count: u64,
|
||||
}
|
||||
|
||||
#[derive(Debug, Default, Serialize)]
|
||||
pub struct DurationSummary {
|
||||
pub avg_ms: f64,
|
||||
pub p50_ms: f64,
|
||||
pub p95_ms: f64,
|
||||
}
|
||||
|
||||
#[must_use]
|
||||
pub fn snapshot_default(window: Window) -> Snapshot {
|
||||
let path = default_path();
|
||||
match snapshot(&path, window) {
|
||||
Ok(s) => s,
|
||||
Err(e) => {
|
||||
tracing::warn!(error = ?e, path = %path.display(), "stats: snapshot failed");
|
||||
empty_snapshot(window)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn default_path() -> PathBuf {
|
||||
crate::paths::state_dir().join("hyperhive-turn-stats.sqlite")
|
||||
}
|
||||
|
||||
fn empty_snapshot(window: Window) -> Snapshot {
|
||||
let now = now_secs();
|
||||
let from = now - window.span_secs();
|
||||
let buckets = fill_buckets(from, now, window.bucket_secs(), &HashMap::new());
|
||||
Snapshot {
|
||||
window: window.label(),
|
||||
bucket_seconds: window.bucket_secs(),
|
||||
now,
|
||||
from,
|
||||
turn_count: 0,
|
||||
buckets,
|
||||
tool_breakdown: Vec::new(),
|
||||
wake_mix: Vec::new(),
|
||||
result_mix: Vec::new(),
|
||||
duration_summary: DurationSummary::default(),
|
||||
}
|
||||
}
|
||||
|
||||
fn snapshot(path: &Path, window: Window) -> Result<Snapshot> {
|
||||
// Read-only open so an in-flight writer (the harness's own
|
||||
// turn_stats sink) never blocks us and we can't corrupt the db
|
||||
// via a query bug.
|
||||
let conn = Connection::open_with_flags(path, OpenFlags::SQLITE_OPEN_READ_ONLY)
|
||||
.with_context(|| format!("open {} read-only", path.display()))?;
|
||||
let now = now_secs();
|
||||
let from = now - window.span_secs();
|
||||
let bucket_secs = window.bucket_secs();
|
||||
|
||||
let mut stmt = conn.prepare(
|
||||
"SELECT started_at, duration_ms,
|
||||
input_tokens, output_tokens,
|
||||
cache_read_input_tokens, cache_creation_input_tokens,
|
||||
last_input_tokens,
|
||||
tool_call_breakdown_json,
|
||||
wake_from, result_kind
|
||||
FROM turn_stats
|
||||
WHERE started_at >= ?1
|
||||
ORDER BY started_at ASC",
|
||||
)?;
|
||||
let rows = stmt.query_map([from], |row| {
|
||||
Ok(Row {
|
||||
started_at: row.get(0)?,
|
||||
duration_ms: row.get::<_, i64>(1)?,
|
||||
input_tokens: u64_from_i64(row.get::<_, i64>(2)?),
|
||||
output_tokens: u64_from_i64(row.get::<_, i64>(3)?),
|
||||
cache_read_input_tokens: u64_from_i64(row.get::<_, i64>(4)?),
|
||||
cache_creation_input_tokens: u64_from_i64(row.get::<_, i64>(5)?),
|
||||
last_input_tokens: u64_from_i64(row.get::<_, i64>(6)?),
|
||||
tool_breakdown_json: row.get::<_, Option<String>>(7)?,
|
||||
wake_from: row.get::<_, String>(8)?,
|
||||
result_kind: row.get::<_, String>(9)?,
|
||||
})
|
||||
})?;
|
||||
|
||||
let mut by_bucket: HashMap<i64, BucketAcc> = HashMap::new();
|
||||
let mut tool_totals: HashMap<String, u64> = HashMap::new();
|
||||
let mut wake_totals: HashMap<String, u64> = HashMap::new();
|
||||
let mut result_totals: HashMap<String, u64> = HashMap::new();
|
||||
let mut all_durations: Vec<i64> = Vec::new();
|
||||
let mut turn_count: u64 = 0;
|
||||
|
||||
for r in rows {
|
||||
let r = r?;
|
||||
turn_count += 1;
|
||||
let bucket_ts = (r.started_at / bucket_secs) * bucket_secs;
|
||||
let acc = by_bucket.entry(bucket_ts).or_default();
|
||||
acc.turn_count += 1;
|
||||
acc.durations.push(r.duration_ms.max(0));
|
||||
acc.input_tokens = acc.input_tokens.saturating_add(r.input_tokens);
|
||||
acc.output_tokens = acc.output_tokens.saturating_add(r.output_tokens);
|
||||
acc.cache_read_input_tokens = acc
|
||||
.cache_read_input_tokens
|
||||
.saturating_add(r.cache_read_input_tokens);
|
||||
acc.cache_creation_input_tokens = acc
|
||||
.cache_creation_input_tokens
|
||||
.saturating_add(r.cache_creation_input_tokens);
|
||||
acc.ctx_sum = acc.ctx_sum.saturating_add(r.last_input_tokens);
|
||||
acc.ctx_max = acc.ctx_max.max(r.last_input_tokens);
|
||||
|
||||
all_durations.push(r.duration_ms.max(0));
|
||||
*wake_totals.entry(r.wake_from).or_insert(0) += 1;
|
||||
*result_totals.entry(r.result_kind).or_insert(0) += 1;
|
||||
|
||||
if let Some(json) = r.tool_breakdown_json
|
||||
&& let Ok(map) = serde_json::from_str::<HashMap<String, u64>>(&json)
|
||||
{
|
||||
for (k, v) in map {
|
||||
*tool_totals.entry(k).or_insert(0) += v;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let buckets = fill_buckets(from, now, bucket_secs, &by_bucket);
|
||||
let duration_summary = summarize_durations(&mut all_durations);
|
||||
|
||||
Ok(Snapshot {
|
||||
window: window.label(),
|
||||
bucket_seconds: bucket_secs,
|
||||
now,
|
||||
from,
|
||||
turn_count,
|
||||
buckets,
|
||||
tool_breakdown: top_n(tool_totals, 10),
|
||||
wake_mix: top_n(wake_totals, 20),
|
||||
result_mix: top_n(result_totals, 20),
|
||||
duration_summary,
|
||||
})
|
||||
}
|
||||
|
||||
struct Row {
|
||||
started_at: i64,
|
||||
duration_ms: i64,
|
||||
input_tokens: u64,
|
||||
output_tokens: u64,
|
||||
cache_read_input_tokens: u64,
|
||||
cache_creation_input_tokens: u64,
|
||||
last_input_tokens: u64,
|
||||
tool_breakdown_json: Option<String>,
|
||||
wake_from: String,
|
||||
result_kind: String,
|
||||
}
|
||||
|
||||
#[derive(Default)]
|
||||
struct BucketAcc {
|
||||
turn_count: u64,
|
||||
durations: Vec<i64>,
|
||||
input_tokens: u64,
|
||||
output_tokens: u64,
|
||||
cache_read_input_tokens: u64,
|
||||
cache_creation_input_tokens: u64,
|
||||
ctx_sum: u64,
|
||||
ctx_max: u64,
|
||||
}
|
||||
|
||||
fn fill_buckets(
|
||||
from: i64,
|
||||
now: i64,
|
||||
bucket_secs: i64,
|
||||
by_bucket: &HashMap<i64, BucketAcc>,
|
||||
) -> Vec<Bucket> {
|
||||
let start = (from / bucket_secs) * bucket_secs;
|
||||
let mut out = Vec::new();
|
||||
let mut ts = start;
|
||||
while ts <= now {
|
||||
let bucket = if let Some(acc) = by_bucket.get(&ts) {
|
||||
let mut sorted = acc.durations.clone();
|
||||
sorted.sort_unstable();
|
||||
let avg = if sorted.is_empty() {
|
||||
0.0
|
||||
} else {
|
||||
#[allow(clippy::cast_precision_loss)]
|
||||
let sum_f = sorted.iter().sum::<i64>() as f64;
|
||||
#[allow(clippy::cast_precision_loss)]
|
||||
let len_f = sorted.len() as f64;
|
||||
sum_f / len_f
|
||||
};
|
||||
let p50 = percentile(&sorted, 50);
|
||||
let p95 = percentile(&sorted, 95);
|
||||
let avg_ctx = if acc.turn_count == 0 {
|
||||
0.0
|
||||
} else {
|
||||
#[allow(clippy::cast_precision_loss)]
|
||||
let sum_f = acc.ctx_sum as f64;
|
||||
#[allow(clippy::cast_precision_loss)]
|
||||
let cnt_f = acc.turn_count as f64;
|
||||
sum_f / cnt_f
|
||||
};
|
||||
Bucket {
|
||||
ts,
|
||||
turn_count: acc.turn_count,
|
||||
avg_duration_ms: avg,
|
||||
p50_duration_ms: p50,
|
||||
p95_duration_ms: p95,
|
||||
input_tokens: acc.input_tokens,
|
||||
output_tokens: acc.output_tokens,
|
||||
cache_read_input_tokens: acc.cache_read_input_tokens,
|
||||
cache_creation_input_tokens: acc.cache_creation_input_tokens,
|
||||
avg_ctx_tokens: avg_ctx,
|
||||
max_ctx_tokens: acc.ctx_max,
|
||||
}
|
||||
} else {
|
||||
Bucket {
|
||||
ts,
|
||||
turn_count: 0,
|
||||
avg_duration_ms: 0.0,
|
||||
p50_duration_ms: 0.0,
|
||||
p95_duration_ms: 0.0,
|
||||
input_tokens: 0,
|
||||
output_tokens: 0,
|
||||
cache_read_input_tokens: 0,
|
||||
cache_creation_input_tokens: 0,
|
||||
avg_ctx_tokens: 0.0,
|
||||
max_ctx_tokens: 0,
|
||||
}
|
||||
};
|
||||
out.push(bucket);
|
||||
ts += bucket_secs;
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
fn summarize_durations(all: &mut [i64]) -> DurationSummary {
|
||||
if all.is_empty() {
|
||||
return DurationSummary::default();
|
||||
}
|
||||
all.sort_unstable();
|
||||
#[allow(clippy::cast_precision_loss)]
|
||||
let sum_f = all.iter().sum::<i64>() as f64;
|
||||
#[allow(clippy::cast_precision_loss)]
|
||||
let len_f = all.len() as f64;
|
||||
DurationSummary {
|
||||
avg_ms: sum_f / len_f,
|
||||
p50_ms: percentile(all, 50),
|
||||
p95_ms: percentile(all, 95),
|
||||
}
|
||||
}
|
||||
|
||||
#[allow(clippy::cast_precision_loss, clippy::cast_possible_truncation, clippy::cast_sign_loss)]
|
||||
fn percentile(sorted: &[i64], pct: u8) -> f64 {
|
||||
if sorted.is_empty() {
|
||||
return 0.0;
|
||||
}
|
||||
if sorted.len() == 1 {
|
||||
return sorted[0] as f64;
|
||||
}
|
||||
// Nearest-rank, clamped.
|
||||
let rank = ((f64::from(pct) / 100.0) * (sorted.len() as f64 - 1.0)).round() as usize;
|
||||
sorted[rank.min(sorted.len() - 1)] as f64
|
||||
}
|
||||
|
||||
fn top_n(map: HashMap<String, u64>, n: usize) -> Vec<KeyCount> {
|
||||
let mut v: Vec<KeyCount> = map
|
||||
.into_iter()
|
||||
.map(|(key, count)| KeyCount { key, count })
|
||||
.collect();
|
||||
v.sort_unstable_by(|a, b| b.count.cmp(&a.count).then_with(|| a.key.cmp(&b.key)));
|
||||
v.truncate(n);
|
||||
v
|
||||
}
|
||||
|
||||
fn u64_from_i64(v: i64) -> u64 {
|
||||
u64::try_from(v).unwrap_or(0)
|
||||
}
|
||||
|
||||
fn now_secs() -> i64 {
|
||||
std::time::SystemTime::now()
|
||||
.duration_since(std::time::UNIX_EPOCH)
|
||||
.map_or(0, |d| i64::try_from(d.as_secs()).unwrap_or(i64::MAX))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use rusqlite::params;
|
||||
use std::sync::atomic::{AtomicU32, Ordering};
|
||||
|
||||
static SEQ: AtomicU32 = AtomicU32::new(0);
|
||||
|
||||
fn tmp_db() -> PathBuf {
|
||||
let n = SEQ.fetch_add(1, Ordering::SeqCst);
|
||||
let pid = std::process::id();
|
||||
std::env::temp_dir().join(format!("hyperhive-stats-test-{pid}-{n}.sqlite"))
|
||||
}
|
||||
|
||||
fn seed_db(path: &Path, rows: &[(i64, i64, &str, &str, &str)]) {
|
||||
let conn = Connection::open(path).unwrap();
|
||||
conn.execute_batch(
|
||||
"CREATE TABLE turn_stats (
|
||||
id INTEGER PRIMARY KEY,
|
||||
started_at INTEGER NOT NULL,
|
||||
ended_at INTEGER NOT NULL,
|
||||
duration_ms INTEGER NOT NULL,
|
||||
model TEXT NOT NULL,
|
||||
wake_from TEXT NOT NULL,
|
||||
input_tokens INTEGER NOT NULL DEFAULT 0,
|
||||
output_tokens INTEGER NOT NULL DEFAULT 0,
|
||||
cache_read_input_tokens INTEGER NOT NULL DEFAULT 0,
|
||||
cache_creation_input_tokens INTEGER NOT NULL DEFAULT 0,
|
||||
last_input_tokens INTEGER NOT NULL DEFAULT 0,
|
||||
last_output_tokens INTEGER NOT NULL DEFAULT 0,
|
||||
last_cache_read_input_tokens INTEGER NOT NULL DEFAULT 0,
|
||||
last_cache_creation_input_tokens INTEGER NOT NULL DEFAULT 0,
|
||||
tool_call_count INTEGER NOT NULL DEFAULT 0,
|
||||
tool_call_breakdown_json TEXT,
|
||||
open_threads_count INTEGER,
|
||||
open_reminders_count INTEGER,
|
||||
result_kind TEXT NOT NULL,
|
||||
note TEXT
|
||||
);",
|
||||
)
|
||||
.unwrap();
|
||||
for (started, dur, wake, result, tools_json) in rows {
|
||||
conn.execute(
|
||||
"INSERT INTO turn_stats
|
||||
(started_at, ended_at, duration_ms, model, wake_from,
|
||||
last_input_tokens, tool_call_breakdown_json, result_kind)
|
||||
VALUES (?1, ?2, ?3, 'm', ?4, 1000, ?5, ?6)",
|
||||
params![started, started + dur / 1000, dur, wake, tools_json, result],
|
||||
)
|
||||
.unwrap();
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn snapshot_aggregates_rows() {
|
||||
let db = tmp_db();
|
||||
let _ = std::fs::remove_file(&db);
|
||||
let now = now_secs();
|
||||
seed_db(
|
||||
&db,
|
||||
&[
|
||||
(now - 600, 5_000, "recv", "ok", r#"{"Read":2,"Bash":1}"#),
|
||||
(now - 300, 10_000, "recv", "ok", r#"{"Read":3}"#),
|
||||
(now - 100, 20_000, "operator", "failed", "{}"),
|
||||
],
|
||||
);
|
||||
let s = snapshot(&db, Window::Day).unwrap();
|
||||
assert_eq!(s.turn_count, 3);
|
||||
assert_eq!(s.window, "24h");
|
||||
assert_eq!(s.bucket_seconds, 3600);
|
||||
let tool_map: HashMap<_, _> = s
|
||||
.tool_breakdown
|
||||
.iter()
|
||||
.map(|kc| (kc.key.clone(), kc.count))
|
||||
.collect();
|
||||
assert_eq!(tool_map.get("Read").copied(), Some(5));
|
||||
assert_eq!(tool_map.get("Bash").copied(), Some(1));
|
||||
let wake_map: HashMap<_, _> = s
|
||||
.wake_mix
|
||||
.iter()
|
||||
.map(|kc| (kc.key.clone(), kc.count))
|
||||
.collect();
|
||||
assert_eq!(wake_map.get("recv").copied(), Some(2));
|
||||
assert_eq!(wake_map.get("operator").copied(), Some(1));
|
||||
let result_map: HashMap<_, _> = s
|
||||
.result_mix
|
||||
.iter()
|
||||
.map(|kc| (kc.key.clone(), kc.count))
|
||||
.collect();
|
||||
assert_eq!(result_map.get("ok").copied(), Some(2));
|
||||
assert_eq!(result_map.get("failed").copied(), Some(1));
|
||||
// Durations: [5000, 10000, 20000] → avg ≈ 11666.67, p50 = 10000, p95 ~ 20000
|
||||
assert!((s.duration_summary.avg_ms - 11_666.666_666_666_666).abs() < 1.0);
|
||||
assert!((s.duration_summary.p50_ms - 10_000.0).abs() < 1.0);
|
||||
assert!((s.duration_summary.p95_ms - 20_000.0).abs() < 1.0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn empty_window_still_paints_buckets() {
|
||||
let db = tmp_db();
|
||||
let _ = std::fs::remove_file(&db);
|
||||
seed_db(&db, &[]);
|
||||
let s = snapshot(&db, Window::Day).unwrap();
|
||||
assert_eq!(s.turn_count, 0);
|
||||
// 24h / 1h buckets = ~24-25 buckets covering the window.
|
||||
assert!(s.buckets.len() >= 24);
|
||||
assert!(s.buckets.iter().all(|b| b.turn_count == 0));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn week_uses_daily_buckets() {
|
||||
let db = tmp_db();
|
||||
let _ = std::fs::remove_file(&db);
|
||||
seed_db(&db, &[]);
|
||||
let s = snapshot(&db, Window::Week).unwrap();
|
||||
assert_eq!(s.window, "7d");
|
||||
assert_eq!(s.bucket_seconds, 86_400);
|
||||
assert!(s.buckets.len() >= 7);
|
||||
}
|
||||
}
|
||||
|
|
@ -107,6 +107,9 @@ pub async fn serve(
|
|||
.route("/api/model", post(post_set_model))
|
||||
.route("/api/new-session", post(post_new_session))
|
||||
.route("/api/loose-ends", get(api_loose_ends))
|
||||
.route("/stats", get(serve_stats))
|
||||
.route("/static/stats.js", get(serve_stats_js))
|
||||
.route("/api/stats", get(api_stats))
|
||||
.with_state(state);
|
||||
let addr = SocketAddr::from(([0, 0, 0, 0], port));
|
||||
let listener = bind_with_retry(addr, "web UI").await?;
|
||||
|
|
@ -198,6 +201,33 @@ async fn serve_marked_js() -> impl IntoResponse {
|
|||
)
|
||||
}
|
||||
|
||||
async fn serve_stats() -> impl IntoResponse {
|
||||
(
|
||||
[("content-type", "text/html; charset=utf-8")],
|
||||
include_str!("../assets/stats.html"),
|
||||
)
|
||||
}
|
||||
|
||||
async fn serve_stats_js() -> impl IntoResponse {
|
||||
(
|
||||
[("content-type", "application/javascript")],
|
||||
include_str!("../assets/stats.js"),
|
||||
)
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct StatsQuery {
|
||||
window: Option<String>,
|
||||
}
|
||||
|
||||
async fn api_stats(
|
||||
State(_state): State<AppState>,
|
||||
axum::extract::Query(q): axum::extract::Query<StatsQuery>,
|
||||
) -> axum::Json<crate::stats::Snapshot> {
|
||||
let window = crate::stats::Window::parse(q.window.as_deref().unwrap_or("24h"));
|
||||
axum::Json(crate::stats::snapshot_default(window))
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
struct StateSnapshot {
|
||||
/// Bus seq at the moment this snapshot was assembled. Clients dedupe
|
||||
|
|
|
|||
|
|
@ -487,6 +487,13 @@
|
|||
el('a', { class: 'name', href: url, target: '_blank', rel: 'noopener' }, c.name),
|
||||
el('span', { class: c.is_manager ? 'role role-m1nd' : 'role role-ag3nt' },
|
||||
c.is_manager ? 'm1nd' : 'ag3nt'),
|
||||
el('a', {
|
||||
class: 'meta',
|
||||
href: url + 'stats',
|
||||
target: '_blank',
|
||||
rel: 'noopener',
|
||||
title: 'per-agent stats page (turn rate, durations, tokens, tool mix)',
|
||||
}, '📊'),
|
||||
);
|
||||
if (pending) {
|
||||
head.append(el('span', { class: 'pending-state' },
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue