hive-agent: keep the last turn's context size in the harness state file and drop the turn sqlite writes
This commit is contained in:
parent
7bd93ecfde
commit
5e5ff64b3c
21 changed files with 261 additions and 629 deletions
|
|
@ -63,8 +63,7 @@ Per-hive (host + in-container):
|
||||||
`/run/hive/priv.sock`; the few root ops (bind-mount edits, nsenter)
|
`/run/hive/priv.sock`; the few root ops (bind-mount edits, nsenter)
|
||||||
`hive-c0re` delegates. See `docs/2-handbook/hives/boundary.md`.
|
`hive-c0re` delegates. See `docs/2-handbook/hives/boundary.md`.
|
||||||
- **`hive-agent/`** — the in-container serve loop: turn-loop *policy*
|
- **`hive-agent/`** — the in-container serve loop: turn-loop *policy*
|
||||||
(`turn.rs`) over `hive-runtime`, per-agent web UI, event + turn-stats
|
(`turn.rs`) over `hive-runtime`, per-agent web UI, the event sink, system-prompt renderer, claude launch config (`mcp_config.rs`).
|
||||||
sinks, system-prompt renderer, claude launch config (`mcp_config.rs`).
|
|
||||||
With `hive-agent-mcp` it's what "hive-ag3nt" refers to — there is no
|
With `hive-agent-mcp` it's what "hive-ag3nt" refers to — there is no
|
||||||
such directory.
|
such directory.
|
||||||
- **`hive-agent-mcp/`** — the embedded MCP server (`hive-mcp-http` unit).
|
- **`hive-agent-mcp/`** — the embedded MCP server (`hive-mcp-http` unit).
|
||||||
|
|
|
||||||
|
|
@ -43,32 +43,19 @@ Retention:
|
||||||
|
|
||||||
### `/harness/hyperhive-turn-stats.sqlite` (per agent)
|
### `/harness/hyperhive-turn-stats.sqlite` (per agent)
|
||||||
|
|
||||||
Per-turn analytics sink. One row per claude turn captures
|
The `hive-bash-daemon` writes a `bash_commands(ts INTEGER, head TEXT)`
|
||||||
identity (`model`, `wake_from`, `result_kind`), timing
|
table in it: one row per executed bash task recording the normalised command head -
|
||||||
(`started_at`, `ended_at`, `duration_ms`), cost (input / output /
|
|
||||||
cache_read / cache_creation token counts), behaviour
|
|
||||||
(`tool_call_count` + `tool_call_breakdown_json`), and post-turn
|
|
||||||
snapshot metrics (`open_threads_count`,
|
|
||||||
`open_reminders_count` — fetched via the same socket the harness
|
|
||||||
already uses for `GetOpenThreads` + `CountPendingReminders`).
|
|
||||||
Bin-loop helpers `build_row` + `record` land each row at
|
|
||||||
`turn_end`; writes are best-effort, a sqlite hiccup logs + lets
|
|
||||||
the turn loop continue.
|
|
||||||
|
|
||||||
The `hive-bash-daemon` (not the harness) writes a sibling
|
|
||||||
`bash_commands(ts INTEGER, head TEXT)` table in the same
|
|
||||||
file: one row per executed bash task recording the normalised command head -
|
|
||||||
the basename of the first real command, looking past `cd repo &&`
|
the basename of the first real command, looking past `cd repo &&`
|
||||||
prefixes, env-assignments, and prefix-runners like `sudo`/`env`. It
|
prefixes, env-assignments, and prefix-runners like `sudo`/`env`. It
|
||||||
feeds the bash-command counter in the agent's OTEL metrics. Best-effort and created on first write
|
feeds the bash-command counter in the agent's OTEL metrics. Best-effort and created on first write
|
||||||
(`CREATE TABLE IF NOT EXISTS`), so it's absent until a bash
|
(`CREATE TABLE IF NOT EXISTS`), so it's absent until a bash
|
||||||
task runs.
|
task runs.
|
||||||
|
|
||||||
turn-stats.sqlite has **no vacuum** — it's one small row per turn
|
The file has **no vacuum**. Older harnesses also wrote a per-turn `turn_stats`
|
||||||
(~hundreds of KB even over months), so pruning it would only lose trend
|
table and a `sessions` table there; nothing writes them now and nothing in the
|
||||||
history for no space gain. The agent has no page that charts it; the
|
harness reads them, but they stay in existing files, where the
|
||||||
[legacy stats viewer](https://forge.constellation.darkest.space/hyperhive/legacy-stats-viewer)
|
[legacy stats viewer](https://forge.constellation.darkest.space/hyperhive/legacy-stats-viewer)
|
||||||
turns the file into the old stats page.
|
can still turn them into the old stats page.
|
||||||
|
|
||||||
### `/state/hyperhive-harness.json` (per agent)
|
### `/state/hyperhive-harness.json` (per agent)
|
||||||
|
|
||||||
|
|
@ -77,7 +64,7 @@ Consolidated harness state file written atomically (`.tmp` + rename) by
|
||||||
Shape:
|
Shape:
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{ "rate_limited": false, "needs_login": false }
|
{ "rate_limited": false, "needs_login": false, "context_tokens": 120000 }
|
||||||
```
|
```
|
||||||
|
|
||||||
- `rate_limited` — set when the harness detects a 429 from the Claude
|
- `rate_limited` — set when the harness detects a 429 from the Claude
|
||||||
|
|
@ -86,6 +73,12 @@ Shape:
|
||||||
- `needs_login` — set when a turn hits 401 (expired OAuth credentials);
|
- `needs_login` — set when a turn hits 401 (expired OAuth credentials);
|
||||||
cleared by `"online"` status (re-auth completed). Drives the
|
cleared by `"online"` status (re-auth completed). Drives the
|
||||||
`needs_login` flag alongside the `claude_has_session` check.
|
`needs_login` flag alongside the `claude_has_session` check.
|
||||||
|
- `context_tokens` — the context size the last turn left, which is what warming
|
||||||
|
the cache costs. The harness overwrites it after every turn and reads it at
|
||||||
|
boot (`Bus::seed_usage`), so the context gauge, the swarm state and the
|
||||||
|
autoreset check start from it instead of from nothing. Absent until a turn has
|
||||||
|
run, so the first boot after an upgrade starts with an empty gauge, as a new
|
||||||
|
agent does.
|
||||||
|
|
||||||
The turn loop is the only writer today, but it still goes
|
The turn loop is the only writer today, but it still goes
|
||||||
read-modify-write under a shared in-process lock and merges into the
|
read-modify-write under a shared in-process lock and merges into the
|
||||||
|
|
@ -235,7 +228,7 @@ Under `/var/lib/hyperhive/agents/<name>/`:
|
||||||
swept — same "audit trail, not cache" treatment as the c0re-side
|
swept — same "audit trail, not cache" treatment as the c0re-side
|
||||||
tables above.
|
tables above.
|
||||||
|
|
||||||
- `hyperhive-turn-stats.sqlite` — per-turn timing stats.
|
- `hyperhive-turn-stats.sqlite` — the bash command heads the bash daemon records.
|
||||||
- `hyperhive-model` / `hyperhive-effort` — single-line model name and
|
- `hyperhive-model` / `hyperhive-effort` — single-line model name and
|
||||||
effort level files, read-only fallbacks.
|
effort level files, read-only fallbacks.
|
||||||
|
|
||||||
|
|
@ -457,7 +450,7 @@ This is the **only** cross-agent mount. An agent without that permission
|
||||||
sees its own dirs and nothing else.
|
sees its own dirs and nothing else.
|
||||||
|
|
||||||
**`harness` isn't mounted at all.** It holds that agent's own runtime
|
**`harness` isn't mounted at all.** It holds that agent's own runtime
|
||||||
material - `bash-tasks/`, the turn-stats and event sqlite dbs - and
|
material - `bash-tasks/`, the sqlite dbs - and
|
||||||
nothing argues for anyone else reading it, let alone writing it.
|
nothing argues for anyone else reading it, let alone writing it.
|
||||||
hive-c0re reads a harness dir **directly on the host** when it wants
|
hive-c0re reads a harness dir **directly on the host** when it wants
|
||||||
those stats, which needs no mount into another container.
|
those stats, which needs no mount into another container.
|
||||||
|
|
|
||||||
|
|
@ -140,7 +140,7 @@ check never reveals _which_ token matched, and a receiver reads request metadata
|
||||||
rather than the claims it authenticated with.
|
rather than the claims it authenticated with.
|
||||||
|
|
||||||
If you need per-agent numbers you can act on, take them from the agent's own
|
If you need per-agent numbers you can act on, take them from the agent's own
|
||||||
turn-stats rather than from a metric label.
|
harness metrics rather than from a metric label.
|
||||||
|
|
||||||
### Authenticated ingest
|
### Authenticated ingest
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -112,7 +112,7 @@ doc for the trait shape.
|
||||||
`HIVE_LABEL` (default `"hive"` for standalone runs; the meta
|
`HIVE_LABEL` (default `"hive"` for standalone runs; the meta
|
||||||
flake sets it unconditionally for any container-deployed agent;
|
flake sets it unconditionally for any container-deployed agent;
|
||||||
see `docs/5-project/process/conventions.md::Hive identity` for the env stack),
|
see `docs/5-project/process/conventions.md::Hive identity` for the env stack),
|
||||||
opens turn-stats sqlite, prepares the on-boot files (see
|
opens the usage sqlite, prepares the on-boot files (see
|
||||||
[claude-invocation](claude-invocation.md#on-boot-files)),
|
[claude-invocation](claude-invocation.md#on-boot-files)),
|
||||||
installs claude plugins, spawns `web_ui::serve` + `vacuum::run`,
|
installs claude plugins, spawns `web_ui::serve` + `vacuum::run`,
|
||||||
and either drops into `serve_loop` directly (`Online`) or parks on
|
and either drops into `serve_loop` directly (`Online`) or parks on
|
||||||
|
|
|
||||||
|
|
@ -115,7 +115,7 @@ pub enum Request {
|
||||||
/// fires.
|
/// fires.
|
||||||
CancelReminder { id: i64 },
|
CancelReminder { id: i64 },
|
||||||
/// This agent's pending-reminder count — used by the pre-`remind` cap
|
/// This agent's pending-reminder count — used by the pre-`remind` cap
|
||||||
/// check and the harness's own turn-stats sink.
|
/// check.
|
||||||
CountPendingReminders,
|
CountPendingReminders,
|
||||||
/// Agent self-service request to compact the current session, mirroring
|
/// Agent self-service request to compact the current session, mirroring
|
||||||
/// the `compact` MCP tool. Gated server-side: only
|
/// the `compact` MCP tool. Gated server-side: only
|
||||||
|
|
|
||||||
|
|
@ -26,7 +26,7 @@ map of the module tree.
|
||||||
`hive-core-agent-sock` wire types.
|
`hive-core-agent-sock` wire types.
|
||||||
- **`turn.rs`** — the turn-loop policy layer: renders the system
|
- **`turn.rs`** — the turn-loop policy layer: renders the system
|
||||||
prompt + MCP config, runs the turn through `hive-runtime`, classifies
|
prompt + MCP config, runs the turn through `hive-runtime`, classifies
|
||||||
the outcome, and feeds the event/turn-stats sinks.
|
the outcome, and feeds the event sink and the OTEL turn metrics.
|
||||||
- **`login.rs`** — the Online or NeedsLogin state at boot, and the wait
|
- **`login.rs`** — the Online or NeedsLogin state at boot, and the wait
|
||||||
for credentials to arrive (the swarm login stores them).
|
for credentials to arrive (the swarm login stores them).
|
||||||
- **`mcp_config.rs`** — renders the per-turn `--mcp-config` /
|
- **`mcp_config.rs`** — renders the per-turn `--mcp-config` /
|
||||||
|
|
@ -36,9 +36,9 @@ map of the module tree.
|
||||||
extra MCP daemons + `hive-agent-mcp` dial into for todo/reminder ops.
|
extra MCP daemons + `hive-agent-mcp` dial into for todo/reminder ops.
|
||||||
- **`vacuum.rs`** — periodic sqlite vacuum sweep for the harness-local
|
- **`vacuum.rs`** — periodic sqlite vacuum sweep for the harness-local
|
||||||
stores.
|
stores.
|
||||||
- **`events.rs` / `turn_stats.rs` / `stats.rs`** — append-only event
|
- **`events.rs` / `stats.rs`** — the event bus and sink, and the read of
|
||||||
sink and per-turn telemetry recording (context usage, cost, tool
|
the bash daemon's command heads for the OTEL bash counter under
|
||||||
favorites) under `harness/`.
|
`harness/`.
|
||||||
- **`prompt.rs`** — system-prompt renderer (persona + tool docs +
|
- **`prompt.rs`** — system-prompt renderer (persona + tool docs +
|
||||||
environment facts).
|
environment facts).
|
||||||
- **`web_ui/`** — the per-agent dashboard (terminal pane, status,
|
- **`web_ui/`** — the per-agent dashboard (terminal pane, status,
|
||||||
|
|
|
||||||
|
|
@ -183,16 +183,11 @@ pub struct Bus {
|
||||||
/// by the serve loop when it builds the session; unset on claude, whose
|
/// by the serve loop when it builds the session; unset on claude, whose
|
||||||
/// turn the swarm UI's cancel stops by signalling the `claude` process.
|
/// turn the swarm UI's cancel stops by signalling the `claude` process.
|
||||||
turn_canceller: Arc<OnceLock<hive_runtime::Canceller>>,
|
turn_canceller: Arc<OnceLock<hive_runtime::Canceller>>,
|
||||||
/// Current fresh-claude-session id (FK to `sessions.id`). Set by the
|
|
||||||
/// bin loop after minting a session row on a fresh start; stamped onto
|
|
||||||
/// every `turn_stats` row until the next fresh session. `None` before
|
|
||||||
/// the first fresh turn or when the stats db is unavailable.
|
|
||||||
session_id: Arc<Mutex<Option<i64>>>,
|
|
||||||
/// One-shot: `run_claude` flips this true when it creates a fresh claude
|
/// One-shot: `run_claude` flips this true when it creates a fresh claude
|
||||||
/// session (`--name <title>` on a title miss). The bin loop takes-and-
|
/// session (`--name <title>` on a title miss). The bin loop takes-and-
|
||||||
/// clears it after the turn to decide whether to mint a new `sessions`
|
/// clears it after the turn for the OTEL session-boundary counter. Session
|
||||||
/// row. Session identity itself is handled entirely in `turn.rs` via the
|
/// identity itself is handled entirely in `turn.rs` via the constant title
|
||||||
/// constant title + archive — this flag is purely a stats signal.
|
/// + archive — this flag is purely a metrics signal.
|
||||||
fresh_session: Arc<AtomicBool>,
|
fresh_session: Arc<AtomicBool>,
|
||||||
/// Per-turn tool-call counter. Reset by the bin loop between
|
/// Per-turn tool-call counter. Reset by the bin loop between
|
||||||
/// turns via `take_tool_calls`. Populated by `observe_stream` as
|
/// turns via `take_tool_calls`. Populated by `observe_stream` as
|
||||||
|
|
@ -260,7 +255,6 @@ impl Bus {
|
||||||
compact_pending: Arc::new(Mutex::new(None)),
|
compact_pending: Arc::new(Mutex::new(None)),
|
||||||
post_compact_wake: Arc::new(Mutex::new(None)),
|
post_compact_wake: Arc::new(Mutex::new(None)),
|
||||||
turn_canceller: Arc::default(),
|
turn_canceller: Arc::default(),
|
||||||
session_id: Arc::new(Mutex::new(None)),
|
|
||||||
fresh_session: Arc::new(AtomicBool::new(false)),
|
fresh_session: Arc::new(AtomicBool::new(false)),
|
||||||
tool_calls: Arc::new(Mutex::new(std::collections::HashMap::new())),
|
tool_calls: Arc::new(Mutex::new(std::collections::HashMap::new())),
|
||||||
last_turn_ended_unix: Arc::new(AtomicI64::new(0)),
|
last_turn_ended_unix: Arc::new(AtomicI64::new(0)),
|
||||||
|
|
@ -347,32 +341,12 @@ impl Bus {
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Take + clear the fresh-session one-shot. The bin loop calls this
|
/// Take + clear the fresh-session one-shot. The bin loop calls this
|
||||||
/// after the turn to decide whether to mint a new `sessions` row.
|
/// after the turn for the OTEL session-boundary counter.
|
||||||
#[must_use]
|
#[must_use]
|
||||||
pub fn take_fresh_session(&self) -> bool {
|
pub fn take_fresh_session(&self) -> bool {
|
||||||
self.fresh_session.swap(false, Ordering::SeqCst)
|
self.fresh_session.swap(false, Ordering::SeqCst)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Currently-active session id (FK to `sessions.id`), or `None`.
|
|
||||||
///
|
|
||||||
/// # Panics
|
|
||||||
///
|
|
||||||
/// Panics if the internal lock is poisoned.
|
|
||||||
#[must_use]
|
|
||||||
pub fn current_session_id(&self) -> Option<i64> {
|
|
||||||
*self.session_id.lock().unwrap()
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Set the active session id after minting a `sessions` row on a
|
|
||||||
/// fresh start.
|
|
||||||
///
|
|
||||||
/// # Panics
|
|
||||||
///
|
|
||||||
/// Panics if the internal lock is poisoned.
|
|
||||||
pub fn set_session_id(&self, id: Option<i64>) {
|
|
||||||
*self.session_id.lock().unwrap() = id;
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Currently-selected claude model name. Read on every turn so a
|
/// Currently-selected claude model name. Read on every turn so a
|
||||||
/// swarm change takes effect on the next turn.
|
/// swarm change takes effect on the next turn.
|
||||||
///
|
///
|
||||||
|
|
@ -445,8 +419,9 @@ impl Bus {
|
||||||
|
|
||||||
/// Seed `last_ctx_usage` + `last_cost_usage` at startup without
|
/// Seed `last_ctx_usage` + `last_cost_usage` at startup without
|
||||||
/// emitting a SSE event. Used by the bin entrypoints to backfill
|
/// emitting a SSE event. Used by the bin entrypoints to backfill
|
||||||
/// from the most recent `turn_stats` row so the per-agent web UI's
|
/// from the context size the last turn kept in the harness state file
|
||||||
/// ctx + cost badges paint real numbers on cold load.
|
/// (`harness_state::read_context_tokens`), so the ctx figure starts from
|
||||||
|
/// what the last turn saw rather than from nothing.
|
||||||
///
|
///
|
||||||
/// # Panics
|
/// # Panics
|
||||||
///
|
///
|
||||||
|
|
@ -642,7 +617,7 @@ impl Bus {
|
||||||
|
|
||||||
/// Snapshot + clear the per-turn tool-call counter. The harness
|
/// Snapshot + clear the per-turn tool-call counter. The harness
|
||||||
/// calls this between turns to fold the breakdown into a
|
/// calls this between turns to fold the breakdown into a
|
||||||
/// `turn_stats` row, then start the next turn with an empty map.
|
/// OTEL tool counts, then start the next turn with an empty map.
|
||||||
///
|
///
|
||||||
/// # Panics
|
/// # Panics
|
||||||
///
|
///
|
||||||
|
|
|
||||||
|
|
@ -190,6 +190,28 @@ pub(crate) fn write_api_key_mode(enabled: bool) {
|
||||||
write_harness_json(&v);
|
write_harness_json(&v);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Keep `tokens`, the context size the turn that just ended left (what warming
|
||||||
|
/// the cache costs), in the consolidated state file, so a restarted harness
|
||||||
|
/// can seed the context gauge and the auto-reset check before its first turn
|
||||||
|
/// (`Bus::seed_usage`). One value, overwritten each turn.
|
||||||
|
pub(crate) fn write_context_tokens(tokens: u64) {
|
||||||
|
let _guard = HARNESS_JSON_LOCK
|
||||||
|
.lock()
|
||||||
|
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||||
|
let mut v = read_harness_json();
|
||||||
|
v["context_tokens"] = tokens.into();
|
||||||
|
write_harness_json(&v);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The context size [`write_context_tokens`] kept, `None` when none was kept
|
||||||
|
/// or it is zero.
|
||||||
|
pub(crate) fn read_context_tokens() -> Option<u64> {
|
||||||
|
read_harness_json()
|
||||||
|
.get("context_tokens")
|
||||||
|
.and_then(serde_json::Value::as_u64)
|
||||||
|
.filter(|n| *n > 0)
|
||||||
|
}
|
||||||
|
|
||||||
/// Compiled-in fallback model used when neither `HIVE_DEFAULT_MODEL` nor a
|
/// Compiled-in fallback model used when neither `HIVE_DEFAULT_MODEL` nor a
|
||||||
/// `hyperhive-model` file is present.
|
/// `hyperhive-model` file is present.
|
||||||
pub const DEFAULT_MODEL: &str = "haiku";
|
pub const DEFAULT_MODEL: &str = "haiku";
|
||||||
|
|
@ -227,18 +249,14 @@ pub fn configured_effort() -> Option<&'static str> {
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
mod tests {
|
mod tests {
|
||||||
use super::{
|
use super::{
|
||||||
DEFAULT_EFFORT, DEFAULT_MODEL, Source, effort_from, model_from, read_harness_state,
|
DEFAULT_EFFORT, DEFAULT_MODEL, Source, effort_from, model_from, read_context_tokens,
|
||||||
write_harness_state,
|
read_harness_state, write_context_tokens, write_harness_state,
|
||||||
};
|
};
|
||||||
|
|
||||||
/// A state file an older harness wrote, with the `active_model` nothing
|
/// Run `f` with `HYPERHIVE_STATE_DIR` pointing at an empty temp dir.
|
||||||
/// reads any more, neither stops the agent reading its flags nor is lost
|
fn in_state_dir(f: impl FnOnce(&std::path::Path)) {
|
||||||
/// or turned into an error by the next write: the file is read as a JSON
|
// `HYPERHIVE_STATE_DIR` is process-wide: one lock for every test that
|
||||||
/// value, not a typed struct, so a leftover field cannot fail to parse.
|
// sets it.
|
||||||
#[test]
|
|
||||||
fn a_leftover_active_model_is_ignored_and_survives_a_write() {
|
|
||||||
// `HYPERHIVE_STATE_DIR` is process-wide; this is the only test that
|
|
||||||
// sets it, so a lock of its own is enough.
|
|
||||||
static LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(());
|
static LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(());
|
||||||
let _lock = LOCK
|
let _lock = LOCK
|
||||||
.lock()
|
.lock()
|
||||||
|
|
@ -247,19 +265,7 @@ mod tests {
|
||||||
let previous = std::env::var_os("HYPERHIVE_STATE_DIR");
|
let previous = std::env::var_os("HYPERHIVE_STATE_DIR");
|
||||||
// SAFETY: serialised by `LOCK`, and restored below.
|
// SAFETY: serialised by `LOCK`, and restored below.
|
||||||
unsafe { std::env::set_var("HYPERHIVE_STATE_DIR", dir.path()) };
|
unsafe { std::env::set_var("HYPERHIVE_STATE_DIR", dir.path()) };
|
||||||
let file = dir.path().join("hyperhive-harness.json");
|
f(dir.path());
|
||||||
std::fs::write(
|
|
||||||
&file,
|
|
||||||
r#"{"rate_limited":true,"needs_login":false,"active_model":"opus"}"#,
|
|
||||||
)
|
|
||||||
.unwrap();
|
|
||||||
|
|
||||||
let read = read_harness_state();
|
|
||||||
write_harness_state(false, true);
|
|
||||||
let after = read_harness_state();
|
|
||||||
let stored: serde_json::Value =
|
|
||||||
serde_json::from_str(&std::fs::read_to_string(&file).unwrap()).unwrap();
|
|
||||||
|
|
||||||
// SAFETY: still holding `LOCK`.
|
// SAFETY: still holding `LOCK`.
|
||||||
unsafe {
|
unsafe {
|
||||||
match previous {
|
match previous {
|
||||||
|
|
@ -267,11 +273,54 @@ mod tests {
|
||||||
None => std::env::remove_var("HYPERHIVE_STATE_DIR"),
|
None => std::env::remove_var("HYPERHIVE_STATE_DIR"),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
assert_eq!(read, (true, false));
|
}
|
||||||
assert_eq!(after, (false, true));
|
|
||||||
assert_eq!(stored["active_model"], "opus", "the leftover is kept");
|
/// A state file an older harness wrote, with the `active_model` nothing
|
||||||
assert_eq!(stored["rate_limited"], false);
|
/// reads any more, neither stops the agent reading its flags nor is lost
|
||||||
assert_eq!(stored["needs_login"], true);
|
/// or turned into an error by the next write: the file is read as a JSON
|
||||||
|
/// value, not a typed struct, so a leftover field cannot fail to parse.
|
||||||
|
#[test]
|
||||||
|
fn a_leftover_active_model_is_ignored_and_survives_a_write() {
|
||||||
|
in_state_dir(|dir| {
|
||||||
|
let file = dir.join("hyperhive-harness.json");
|
||||||
|
std::fs::write(
|
||||||
|
&file,
|
||||||
|
r#"{"rate_limited":true,"needs_login":false,"active_model":"opus"}"#,
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
|
||||||
|
let read = read_harness_state();
|
||||||
|
write_harness_state(false, true);
|
||||||
|
let after = read_harness_state();
|
||||||
|
let stored: serde_json::Value =
|
||||||
|
serde_json::from_str(&std::fs::read_to_string(&file).unwrap()).unwrap();
|
||||||
|
|
||||||
|
assert_eq!(read, (true, false));
|
||||||
|
assert_eq!(after, (false, true));
|
||||||
|
assert_eq!(stored["active_model"], "opus", "the leftover is kept");
|
||||||
|
assert_eq!(stored["rate_limited"], false);
|
||||||
|
assert_eq!(stored["needs_login"], true);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The context size is one field of the same file: nothing kept reads as
|
||||||
|
/// nothing, the latest turn replaces the one before, a zero is no reading,
|
||||||
|
/// and the flags beside it survive every write.
|
||||||
|
#[test]
|
||||||
|
fn the_context_size_is_one_field_that_the_latest_turn_replaces() {
|
||||||
|
in_state_dir(|_| {
|
||||||
|
assert_eq!(read_context_tokens(), None);
|
||||||
|
write_harness_state(true, false);
|
||||||
|
|
||||||
|
write_context_tokens(100_000);
|
||||||
|
assert_eq!(read_context_tokens(), Some(100_000));
|
||||||
|
write_context_tokens(180_000);
|
||||||
|
assert_eq!(read_context_tokens(), Some(180_000));
|
||||||
|
assert_eq!(read_harness_state(), (true, false));
|
||||||
|
|
||||||
|
write_context_tokens(0);
|
||||||
|
assert_eq!(read_context_tokens(), None);
|
||||||
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
|
|
|
||||||
|
|
@ -31,7 +31,6 @@ mod reminders;
|
||||||
mod router_todos;
|
mod router_todos;
|
||||||
mod schedule_timer;
|
mod schedule_timer;
|
||||||
mod serve_common;
|
mod serve_common;
|
||||||
mod sqlite_writer;
|
|
||||||
mod state_entry_watch;
|
mod state_entry_watch;
|
||||||
mod stats;
|
mod stats;
|
||||||
mod subagent_role_watch;
|
mod subagent_role_watch;
|
||||||
|
|
@ -52,7 +51,6 @@ mod todos;
|
||||||
mod todos_gauge;
|
mod todos_gauge;
|
||||||
mod transcribe;
|
mod transcribe;
|
||||||
mod turn;
|
mod turn;
|
||||||
mod turn_stats;
|
|
||||||
mod vacuum;
|
mod vacuum;
|
||||||
mod web_ui;
|
mod web_ui;
|
||||||
|
|
||||||
|
|
@ -107,7 +105,6 @@ use std::time::Duration;
|
||||||
|
|
||||||
use crate::events::{Bus, LiveEvent, TurnState};
|
use crate::events::{Bus, LiveEvent, TurnState};
|
||||||
use crate::login::LoginState;
|
use crate::login::LoginState;
|
||||||
use crate::turn_stats::TurnStats;
|
|
||||||
use anyhow::Result;
|
use anyhow::Result;
|
||||||
use clap::Parser;
|
use clap::Parser;
|
||||||
use hive_core_agent_sock::{Request, Response};
|
use hive_core_agent_sock::{Request, Response};
|
||||||
|
|
@ -526,12 +523,11 @@ async fn serve_main<S: Surface>(socket: &Path, poll_ms: u64) -> Result<()> {
|
||||||
// `hive_sh4re::inbox::INTERRUPTED_HINT`. Shared between the web server task and
|
// `hive_sh4re::inbox::INTERRUPTED_HINT`. Shared between the web server task and
|
||||||
// the serve loop the same way `bus`/`todo_wake` are.
|
// the serve loop the same way `bus`/`todo_wake` are.
|
||||||
let interrupted = Arc::new(std::sync::atomic::AtomicBool::new(false));
|
let interrupted = Arc::new(std::sync::atomic::AtomicBool::new(false));
|
||||||
let stats = TurnStats::open_default();
|
if let Some(tokens) = harness_state::read_context_tokens() {
|
||||||
if let Some(s) = &stats {
|
// Only the total is kept; it is the one figure anything reads.
|
||||||
let (ctx, cost) = s.last_usage();
|
let mut ctx = hive_claude::TokenUsage::default();
|
||||||
if ctx.is_some() || cost.is_some() {
|
ctx.input_tokens = tokens;
|
||||||
bus.seed_usage(ctx, cost);
|
bus.seed_usage(Some(ctx), None);
|
||||||
}
|
|
||||||
}
|
}
|
||||||
let files = turn::TurnFiles::prepare(socket, &label).await?;
|
let files = turn::TurnFiles::prepare(socket, &label).await?;
|
||||||
plugins::install_configured().await;
|
plugins::install_configured().await;
|
||||||
|
|
@ -612,7 +608,6 @@ async fn serve_main<S: Surface>(socket: &Path, poll_ms: u64) -> Result<()> {
|
||||||
login_state,
|
login_state,
|
||||||
claude_dir,
|
claude_dir,
|
||||||
bus,
|
bus,
|
||||||
stats,
|
|
||||||
&files,
|
&files,
|
||||||
todo_wake,
|
todo_wake,
|
||||||
todos_store,
|
todos_store,
|
||||||
|
|
@ -729,7 +724,6 @@ async fn serve_loop<S: Surface>(
|
||||||
login_state: Arc<Mutex<LoginState>>,
|
login_state: Arc<Mutex<LoginState>>,
|
||||||
claude_dir: std::path::PathBuf,
|
claude_dir: std::path::PathBuf,
|
||||||
bus: Bus,
|
bus: Bus,
|
||||||
stats: Option<TurnStats>,
|
|
||||||
files: &turn::TurnFiles,
|
files: &turn::TurnFiles,
|
||||||
todo_wake: Arc<tokio::sync::Notify>,
|
todo_wake: Arc<tokio::sync::Notify>,
|
||||||
todos_store: Option<Arc<todos::Todos>>,
|
todos_store: Option<Arc<todos::Todos>>,
|
||||||
|
|
@ -863,7 +857,6 @@ async fn serve_loop<S: Surface>(
|
||||||
let _ = handle_turn::<S>(
|
let _ = handle_turn::<S>(
|
||||||
socket,
|
socket,
|
||||||
&bus,
|
&bus,
|
||||||
stats.as_ref(),
|
|
||||||
files,
|
files,
|
||||||
&session,
|
&session,
|
||||||
graceful_stop_message(),
|
graceful_stop_message(),
|
||||||
|
|
@ -877,7 +870,6 @@ async fn serve_loop<S: Surface>(
|
||||||
let turn_ctx = TurnCtx {
|
let turn_ctx = TurnCtx {
|
||||||
socket,
|
socket,
|
||||||
bus: &bus,
|
bus: &bus,
|
||||||
stats: stats.as_ref(),
|
|
||||||
files,
|
files,
|
||||||
session: &session,
|
session: &session,
|
||||||
interrupted: &interrupted,
|
interrupted: &interrupted,
|
||||||
|
|
@ -897,7 +889,6 @@ async fn serve_loop<S: Surface>(
|
||||||
struct TurnCtx<'a> {
|
struct TurnCtx<'a> {
|
||||||
socket: &'a Path,
|
socket: &'a Path,
|
||||||
bus: &'a Bus,
|
bus: &'a Bus,
|
||||||
stats: Option<&'a TurnStats>,
|
|
||||||
files: &'a turn::TurnFiles,
|
files: &'a turn::TurnFiles,
|
||||||
session: &'a turn::AgentSession,
|
session: &'a turn::AgentSession,
|
||||||
interrupted: &'a Arc<std::sync::atomic::AtomicBool>,
|
interrupted: &'a Arc<std::sync::atomic::AtomicBool>,
|
||||||
|
|
@ -923,7 +914,6 @@ async fn drive_turn_and_wake_chain<S: Surface>(
|
||||||
let ctrl = handle_turn::<S>(
|
let ctrl = handle_turn::<S>(
|
||||||
ctx.socket,
|
ctx.socket,
|
||||||
ctx.bus,
|
ctx.bus,
|
||||||
ctx.stats,
|
|
||||||
ctx.files,
|
ctx.files,
|
||||||
ctx.session,
|
ctx.session,
|
||||||
next,
|
next,
|
||||||
|
|
@ -1005,12 +995,11 @@ fn apply_todo_wake_checked(checked: Option<bool>, todo_miss_streak: &mut u32, bu
|
||||||
/// Drive a single turn: emit boot-of-turn events, run claude, ack on
|
/// Drive a single turn: emit boot-of-turn events, run claude, ack on
|
||||||
/// success / requeue on rate-limit-or-401 / log `turn failed` on a hard
|
/// success / requeue on rate-limit-or-401 / log `turn failed` on a hard
|
||||||
/// failure (the line the `hh-agent-turn-failed` Grafana rule alerts on),
|
/// failure (the line the `hh-agent-turn-failed` Grafana rule alerts on),
|
||||||
/// record stats. Returns a `TurnControl` carrying the auth-failed flag and
|
/// keep the context size. Returns a `TurnControl` carrying the auth-failed flag and
|
||||||
/// the todo-wake-checked signal — the serve loop decides what to do next.
|
/// the todo-wake-checked signal — the serve loop decides what to do next.
|
||||||
async fn handle_turn<S: Surface>(
|
async fn handle_turn<S: Surface>(
|
||||||
socket: &Path,
|
socket: &Path,
|
||||||
bus: &Bus,
|
bus: &Bus,
|
||||||
stats: Option<&TurnStats>,
|
|
||||||
files: &turn::TurnFiles,
|
files: &turn::TurnFiles,
|
||||||
session: &turn::AgentSession,
|
session: &turn::AgentSession,
|
||||||
first: hive_sh4re::inbox::DeliveredMessage,
|
first: hive_sh4re::inbox::DeliveredMessage,
|
||||||
|
|
@ -1045,37 +1034,20 @@ async fn handle_turn<S: Surface>(
|
||||||
handle_turn_error_recovery::<S>(&outcome, bus, socket).await;
|
handle_turn_error_recovery::<S>(&outcome, bus, socket).await;
|
||||||
// Single read of the per-turn tool-call counter: `take_tool_calls`
|
// Single read of the per-turn tool-call counter: `take_tool_calls`
|
||||||
// resets it, so this is the only chance to see which tools this turn
|
// resets it, so this is the only chance to see which tools this turn
|
||||||
// invoked. Used both for the stats row below (if stats are configured)
|
// invoked. Used both for the OTEL tool counts below and for the
|
||||||
// and for the todo-miss-streak signal returned to the serve loop —
|
// todo-miss-streak signal returned to the serve loop.
|
||||||
// deliberately unconditional (not gated on `stats.is_some()`) so the
|
|
||||||
// counter is reset every turn regardless, and so a stats-less harness
|
|
||||||
// (tests, or a future config without a stats sink) still gets the
|
|
||||||
// miss-streak signal.
|
|
||||||
let tool_calls = bus.take_tool_calls();
|
let tool_calls = bus.take_tool_calls();
|
||||||
let todo_wake_checked =
|
let todo_wake_checked =
|
||||||
(from == "todo").then(|| tool_calls.contains_key("mcp__hyperhive__get_loose_ends"));
|
(from == "todo").then(|| tool_calls.contains_key("mcp__hyperhive__get_loose_ends"));
|
||||||
// Read-and-clear once per turn unconditionally (same rationale as
|
// Read-and-clear once per turn (same rationale as `take_tool_calls`
|
||||||
// `take_tool_calls` above — the harness gets exactly one chance to
|
// above — the harness gets exactly one chance to observe this flag): the
|
||||||
// observe this flag) rather than only when the sqlite sink happens to be
|
// OTEL turn-metrics exporter's session-boundary counter needs it.
|
||||||
// configured: the OTEL turn-metrics exporter's session-boundary counter
|
|
||||||
// needs it too, independent of `stats`.
|
|
||||||
let fresh_session = bus.take_fresh_session();
|
let fresh_session = bus.take_fresh_session();
|
||||||
if let Some(stats) = stats {
|
|
||||||
// Fresh session this turn → mint a `sessions` row and set its id on
|
|
||||||
// the bus so this turn (and subsequent ones until the next fresh
|
|
||||||
// start) stamp `turn_stats.session_id`.
|
|
||||||
if fresh_session {
|
|
||||||
let sid = stats.start_session(started_at, &model_at_start);
|
|
||||||
bus.set_session_id(sid);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
let ended_at = chrono::Utc::now().timestamp();
|
|
||||||
let duration_ms = i64::try_from(started_instant.elapsed().as_millis()).unwrap_or(i64::MAX);
|
let duration_ms = i64::try_from(started_instant.elapsed().as_millis()).unwrap_or(i64::MAX);
|
||||||
let (open_threads, open_reminders) = S::post_turn_counts(socket).await;
|
let (open_threads, open_reminders) = S::post_turn_counts(socket).await;
|
||||||
let context_window = bus.api_context_window();
|
let context_window = bus.api_context_window();
|
||||||
let row = serve_common::build_row(serve_common::TurnRowArgs {
|
let row = serve_common::build_row(serve_common::TurnRowArgs {
|
||||||
started_at,
|
started_at,
|
||||||
ended_at,
|
|
||||||
duration_ms,
|
duration_ms,
|
||||||
model: model_at_start,
|
model: model_at_start,
|
||||||
wake_from: from.clone(),
|
wake_from: from.clone(),
|
||||||
|
|
@ -1086,8 +1058,8 @@ async fn handle_turn<S: Surface>(
|
||||||
open_reminders_count: open_reminders,
|
open_reminders_count: open_reminders,
|
||||||
});
|
});
|
||||||
// Harness-only OTEL metrics (duration/wake_from/result_kind/loose-ends/
|
// Harness-only OTEL metrics (duration/wake_from/result_kind/loose-ends/
|
||||||
// session boundaries/context fill) — independent of the sqlite sink
|
// session boundaries/context fill), a cheap no-op when OTEL isn't
|
||||||
// below, and a cheap no-op when OTEL isn't configured. See
|
// configured. See
|
||||||
// `otel_turn_metrics`'s module doc for why token/tool-count are
|
// `otel_turn_metrics`'s module doc for why token/tool-count are
|
||||||
// deliberately not here, and why cost is only ever an ACP agent's.
|
// deliberately not here, and why cost is only ever an ACP agent's.
|
||||||
otel_turn_metrics::record(&row, fresh_session);
|
otel_turn_metrics::record(&row, fresh_session);
|
||||||
|
|
@ -1097,8 +1069,8 @@ async fn handle_turn<S: Surface>(
|
||||||
for cost in hive_runtime::Runtime::take_reported_cost(session) {
|
for cost in hive_runtime::Runtime::take_reported_cost(session) {
|
||||||
otel_turn_metrics::record_reported_cost(&row.model, &cost);
|
otel_turn_metrics::record_reported_cost(&row.model, &cost);
|
||||||
}
|
}
|
||||||
if let Some(stats) = stats {
|
if let Some(ctx) = bus.last_ctx_usage() {
|
||||||
stats.record(&row);
|
harness_state::write_context_tokens(ctx.context_tokens());
|
||||||
}
|
}
|
||||||
let pending = S::inbox_unread(socket).await;
|
let pending = S::inbox_unread(socket).await;
|
||||||
if pending > 0 {
|
if pending > 0 {
|
||||||
|
|
|
||||||
|
|
@ -39,7 +39,7 @@ use opentelemetry::metrics::{Counter, Gauge, Histogram, MeterProvider as _};
|
||||||
use opentelemetry_otlp::{MetricExporter, Protocol, WithExportConfig};
|
use opentelemetry_otlp::{MetricExporter, Protocol, WithExportConfig};
|
||||||
use opentelemetry_sdk::metrics::{PeriodicReader, SdkMeterProvider};
|
use opentelemetry_sdk::metrics::{PeriodicReader, SdkMeterProvider};
|
||||||
|
|
||||||
use crate::turn_stats::TurnStatRow;
|
use crate::serve_common::TurnStatRow;
|
||||||
|
|
||||||
/// Export cadence. Matches `hive-c0re`'s container-metrics exporter so the
|
/// Export cadence. Matches `hive-c0re`'s container-metrics exporter so the
|
||||||
/// two batch on the same rhythm.
|
/// two batch on the same rhythm.
|
||||||
|
|
@ -123,7 +123,7 @@ static INSTRUMENTS: OnceLock<Option<Instruments>> = OnceLock::new();
|
||||||
/// `fresh_session` is the harness's own "did this turn start a brand-new
|
/// `fresh_session` is the harness's own "did this turn start a brand-new
|
||||||
/// claude session" flag (`Bus::take_fresh_session`) — pass it through rather
|
/// claude session" flag (`Bus::take_fresh_session`) — pass it through rather
|
||||||
/// than re-deriving it here, since the harness only gets to observe it once
|
/// than re-deriving it here, since the harness only gets to observe it once
|
||||||
/// per turn (the same flag gates minting a `turn_stats` `sessions` row).
|
/// per turn.
|
||||||
pub fn record(row: &TurnStatRow, fresh_session: bool) {
|
pub fn record(row: &TurnStatRow, fresh_session: bool) {
|
||||||
let Some(inst) = INSTRUMENTS.get_or_init(build).as_ref() else {
|
let Some(inst) = INSTRUMENTS.get_or_init(build).as_ref() else {
|
||||||
return;
|
return;
|
||||||
|
|
|
||||||
|
|
@ -42,7 +42,7 @@ pub fn harness_dir() -> PathBuf {
|
||||||
|
|
||||||
/// Consolidated harness-local state db — todos + reminders, one table
|
/// Consolidated harness-local state db — todos + reminders, one table
|
||||||
/// each — mutable per-agent state the harness owns, kept apart
|
/// each — mutable per-agent state the harness owns, kept apart
|
||||||
/// from the append-only turn-stats db. Each store's `open()` only
|
/// from the usage db. Each store's `open()` only
|
||||||
/// applies its own `CREATE TABLE IF NOT EXISTS` against this same path
|
/// applies its own `CREATE TABLE IF NOT EXISTS` against this same path
|
||||||
/// (see their call sites), so distinct table names keep them from
|
/// (see their call sites), so distinct table names keep them from
|
||||||
/// colliding without needing a per-store path wrapper here.
|
/// colliding without needing a per-store path wrapper here.
|
||||||
|
|
|
||||||
|
|
@ -6,7 +6,6 @@
|
||||||
|
|
||||||
use crate::events::Bus;
|
use crate::events::Bus;
|
||||||
use crate::turn::{TurnError, TurnOutcome};
|
use crate::turn::{TurnError, TurnOutcome};
|
||||||
use crate::turn_stats::TurnStatRow;
|
|
||||||
|
|
||||||
/// Assemble the per-turn wake prompt string. The role/tools/etc. live in the
|
/// Assemble the per-turn wake prompt string. The role/tools/etc. live in the
|
||||||
/// system prompt; this is just the wake signal body. `redelivered` prepends a "may already be handled" banner; `interrupted`
|
/// system prompt; this is just the wake signal body. `redelivered` prepends a "may already be handled" banner; `interrupted`
|
||||||
|
|
@ -26,15 +25,33 @@ pub fn format_wake_prompt(from: &str, body: &str, redelivered: bool, interrupted
|
||||||
format!("{banner}Incoming message from `{from}`:\n---\n{body}\n---")
|
format!("{banner}Incoming message from `{from}`:\n---\n{body}\n---")
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Field-named args for [`build_row`]. Mirrors the turn-stats row
|
/// What one finished turn leaves for the OTEL turn metrics
|
||||||
/// columns; `outcome` and `bus` borrow for the duration of the call.
|
/// (`otel_turn_metrics::record`).
|
||||||
/// `tool_calls` is passed in (rather than pulled from `bus` internally)
|
#[derive(Debug, Clone)]
|
||||||
/// because the caller needs the same map to check for a `get_loose_ends`
|
pub struct TurnStatRow {
|
||||||
/// call before it's consumed — `Bus::take_tool_calls` resets the counter,
|
pub started_at: i64,
|
||||||
/// so there's only one chance to read it per turn.
|
pub duration_ms: i64,
|
||||||
|
pub model: String,
|
||||||
|
pub wake_from: String,
|
||||||
|
/// Prompt tokens billed at the full rate, summed over the turn's inferences.
|
||||||
|
pub input_tokens: u64,
|
||||||
|
/// Per-tool breakdown as JSON: `{"Read":12,"Bash":3,...}`; `None` when no
|
||||||
|
/// tool was called.
|
||||||
|
pub tool_call_breakdown_json: Option<String>,
|
||||||
|
/// Post-turn loose-ends snapshot; `None` when the request failed.
|
||||||
|
pub open_threads_count: Option<u64>,
|
||||||
|
pub open_reminders_count: Option<u64>,
|
||||||
|
/// `"ok" | "compacted" | "failed" | ...`, see [`build_row`].
|
||||||
|
pub result_kind: &'static str,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Field-named args for [`build_row`]; `outcome` and `bus` borrow for the
|
||||||
|
/// duration of the call. `tool_calls` is passed in (rather than pulled from
|
||||||
|
/// `bus` internally) because the caller needs the same map to check for a
|
||||||
|
/// `get_loose_ends` call before it's consumed — `Bus::take_tool_calls` resets
|
||||||
|
/// the counter, so there's only one chance to read it per turn.
|
||||||
pub struct TurnRowArgs<'a> {
|
pub struct TurnRowArgs<'a> {
|
||||||
pub started_at: i64,
|
pub started_at: i64,
|
||||||
pub ended_at: i64,
|
|
||||||
pub duration_ms: i64,
|
pub duration_ms: i64,
|
||||||
pub model: String,
|
pub model: String,
|
||||||
pub wake_from: String,
|
pub wake_from: String,
|
||||||
|
|
@ -52,7 +69,6 @@ pub struct TurnRowArgs<'a> {
|
||||||
pub fn build_row(args: TurnRowArgs<'_>) -> TurnStatRow {
|
pub fn build_row(args: TurnRowArgs<'_>) -> TurnStatRow {
|
||||||
let TurnRowArgs {
|
let TurnRowArgs {
|
||||||
started_at,
|
started_at,
|
||||||
ended_at,
|
|
||||||
duration_ms,
|
duration_ms,
|
||||||
model,
|
model,
|
||||||
wake_from,
|
wake_from,
|
||||||
|
|
@ -64,50 +80,36 @@ pub fn build_row(args: TurnRowArgs<'_>) -> TurnStatRow {
|
||||||
} = args;
|
} = args;
|
||||||
// Prefer the API-resolved model id (e.g. `claude-opus-4-8`) captured
|
// Prefer the API-resolved model id (e.g. `claude-opus-4-8`) captured
|
||||||
// from this turn's assistant events over the requested `--model`
|
// from this turn's assistant events over the requested `--model`
|
||||||
// name/alias, so the model-mix + cost rollup label the concrete
|
// name/alias, so the model-mix rollup labels the concrete version that
|
||||||
// version that ran. Falls back to the requested name on a degenerate
|
// ran. Falls back to the requested name on a degenerate turn that
|
||||||
// turn that produced no assistant event.
|
// produced no assistant event.
|
||||||
let model = bus.last_resolved_model().unwrap_or(model);
|
let model = bus.last_resolved_model().unwrap_or(model);
|
||||||
let cost = bus.last_cost_usage().unwrap_or_default();
|
let cost = bus.last_cost_usage().unwrap_or_default();
|
||||||
let ctx = bus.last_ctx_usage().unwrap_or(cost);
|
|
||||||
let tool_call_count: u64 = tool_calls.values().copied().sum();
|
|
||||||
let tool_call_breakdown_json = if tool_calls.is_empty() {
|
let tool_call_breakdown_json = if tool_calls.is_empty() {
|
||||||
None
|
None
|
||||||
} else {
|
} else {
|
||||||
serde_json::to_string(&tool_calls).ok()
|
serde_json::to_string(&tool_calls).ok()
|
||||||
};
|
};
|
||||||
let (result_kind, note) = match outcome {
|
let result_kind = match outcome {
|
||||||
Ok(false) => ("ok", None),
|
Ok(false) => "ok",
|
||||||
Ok(true) => ("compacted", None),
|
Ok(true) => "compacted",
|
||||||
Err(TurnError::PromptTooLong) => ("prompt_too_long", None),
|
Err(TurnError::PromptTooLong) => "prompt_too_long",
|
||||||
Err(TurnError::RateLimited) => ("rate_limited", None),
|
Err(TurnError::RateLimited) => "rate_limited",
|
||||||
Err(TurnError::AuthFailed) => ("auth_failed", None),
|
Err(TurnError::AuthFailed) => "auth_failed",
|
||||||
Err(TurnError::SessionNotFound) => ("session_not_found", None),
|
Err(TurnError::SessionNotFound) => "session_not_found",
|
||||||
Err(TurnError::ApiStall) => ("api_stall", None),
|
Err(TurnError::ApiStall | TurnError::AgentStall(_)) => "api_stall",
|
||||||
Err(TurnError::AgentStall(note)) => ("api_stall", Some(note.clone())),
|
Err(TurnError::Failed(_)) => "failed",
|
||||||
Err(TurnError::Failed(e)) => ("failed", Some(format!("{e:#}"))),
|
|
||||||
};
|
};
|
||||||
TurnStatRow {
|
TurnStatRow {
|
||||||
started_at,
|
started_at,
|
||||||
ended_at,
|
|
||||||
duration_ms,
|
duration_ms,
|
||||||
model,
|
model,
|
||||||
wake_from,
|
wake_from,
|
||||||
input_tokens: cost.input_tokens,
|
input_tokens: cost.input_tokens,
|
||||||
output_tokens: cost.output_tokens,
|
|
||||||
cache_read_input_tokens: cost.cache_read_input_tokens,
|
|
||||||
cache_creation_input_tokens: cost.cache_creation_input_tokens,
|
|
||||||
last_input_tokens: ctx.input_tokens,
|
|
||||||
last_output_tokens: ctx.output_tokens,
|
|
||||||
last_cache_read_input_tokens: ctx.cache_read_input_tokens,
|
|
||||||
last_cache_creation_input_tokens: ctx.cache_creation_input_tokens,
|
|
||||||
tool_call_count,
|
|
||||||
tool_call_breakdown_json,
|
tool_call_breakdown_json,
|
||||||
open_threads_count,
|
open_threads_count,
|
||||||
open_reminders_count,
|
open_reminders_count,
|
||||||
result_kind,
|
result_kind,
|
||||||
note,
|
|
||||||
session_id: bus.current_session_id(),
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -135,3 +137,89 @@ mod wake_prompt_tests {
|
||||||
assert!(plain.starts_with("Incoming message from `a`:"));
|
assert!(plain.starts_with("Incoming message from `a`:"));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod build_row_tests {
|
||||||
|
use super::{TurnRowArgs, build_row};
|
||||||
|
use crate::events::Bus;
|
||||||
|
use crate::turn::TurnError;
|
||||||
|
|
||||||
|
fn row_for(
|
||||||
|
outcome: &crate::turn::TurnOutcome,
|
||||||
|
tool_calls: &[(&str, u64)],
|
||||||
|
) -> super::TurnStatRow {
|
||||||
|
build_row(TurnRowArgs {
|
||||||
|
started_at: 10,
|
||||||
|
duration_ms: 2_000,
|
||||||
|
model: "opus".to_owned(),
|
||||||
|
wake_from: "todo".to_owned(),
|
||||||
|
outcome,
|
||||||
|
bus: &Bus::new(),
|
||||||
|
tool_calls: tool_calls
|
||||||
|
.iter()
|
||||||
|
.map(|(k, v)| ((*k).to_owned(), *v))
|
||||||
|
.collect(),
|
||||||
|
open_threads_count: Some(2),
|
||||||
|
open_reminders_count: None,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn the_outcome_names_the_result_kind() {
|
||||||
|
let _dir = crate::paths::harness_dir_guard();
|
||||||
|
for (outcome, kind) in [
|
||||||
|
(Ok(false), "ok"),
|
||||||
|
(Ok(true), "compacted"),
|
||||||
|
(Err(TurnError::PromptTooLong), "prompt_too_long"),
|
||||||
|
(Err(TurnError::RateLimited), "rate_limited"),
|
||||||
|
(Err(TurnError::AuthFailed), "auth_failed"),
|
||||||
|
(Err(TurnError::SessionNotFound), "session_not_found"),
|
||||||
|
(Err(TurnError::ApiStall), "api_stall"),
|
||||||
|
(Err(TurnError::AgentStall("quiet".into())), "api_stall"),
|
||||||
|
(Err(TurnError::Failed(anyhow::anyhow!("boom"))), "failed"),
|
||||||
|
] {
|
||||||
|
assert_eq!(row_for(&outcome, &[]).result_kind, kind);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn the_row_carries_the_turns_identity_counts_and_tool_breakdown() {
|
||||||
|
let _dir = crate::paths::harness_dir_guard();
|
||||||
|
let row = row_for(&Ok(false), &[("Bash", 3)]);
|
||||||
|
|
||||||
|
assert_eq!((row.started_at, row.duration_ms), (10, 2_000));
|
||||||
|
assert_eq!(
|
||||||
|
(row.model.as_str(), row.wake_from.as_str()),
|
||||||
|
("opus", "todo")
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
(row.open_threads_count, row.open_reminders_count),
|
||||||
|
(Some(2), None)
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
row.tool_call_breakdown_json.as_deref(),
|
||||||
|
Some(r#"{"Bash":3}"#)
|
||||||
|
);
|
||||||
|
assert_eq!(row_for(&Ok(false), &[]).tool_call_breakdown_json, None);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The model the API resolved wins over the requested alias.
|
||||||
|
#[test]
|
||||||
|
fn the_resolved_model_replaces_the_requested_one() {
|
||||||
|
let _dir = crate::paths::harness_dir_guard();
|
||||||
|
let bus = Bus::new();
|
||||||
|
bus.set_resolved_model(Some("claude-opus-4-8".to_owned()));
|
||||||
|
let row = build_row(TurnRowArgs {
|
||||||
|
started_at: 0,
|
||||||
|
duration_ms: 0,
|
||||||
|
model: "opus".to_owned(),
|
||||||
|
wake_from: "todo".to_owned(),
|
||||||
|
outcome: &Ok(false),
|
||||||
|
bus: &bus,
|
||||||
|
tool_calls: std::collections::HashMap::new(),
|
||||||
|
open_threads_count: None,
|
||||||
|
open_reminders_count: None,
|
||||||
|
});
|
||||||
|
assert_eq!(row.model, "claude-opus-4-8");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
|
||||||
|
|
@ -1,10 +0,0 @@
|
||||||
//! Shared connection setup for the harness's own sqlite writers.
|
|
||||||
//!
|
|
||||||
//! A write that overlaps another connection's lock (the hourly vacuum, a
|
|
||||||
//! reader) fails at once with `SQLITE_BUSY` unless the connection has a busy
|
|
||||||
//! timeout, so writers set [`BUSY_TIMEOUT`] to wait instead of dropping the row.
|
|
||||||
|
|
||||||
use std::time::Duration;
|
|
||||||
|
|
||||||
/// How long a writer waits on a locked database before giving up.
|
|
||||||
pub const BUSY_TIMEOUT: Duration = Duration::from_secs(5);
|
|
||||||
|
|
@ -1,4 +1,4 @@
|
||||||
//! The shell-command heads the bash daemon records in the turn-stats database,
|
//! The shell-command heads the bash daemon records in the harness's `hyperhive-turn-stats.sqlite`,
|
||||||
//! read for the OTEL bash-command counter.
|
//! read for the OTEL bash-command counter.
|
||||||
|
|
||||||
use std::path::PathBuf;
|
use std::path::PathBuf;
|
||||||
|
|
|
||||||
|
|
@ -381,8 +381,7 @@ mod tests {
|
||||||
use tokio::time::{Duration, Instant};
|
use tokio::time::{Duration, Instant};
|
||||||
|
|
||||||
/// `TokenUsage` is `#[non_exhaustive]`, so a downstream crate cannot
|
/// `TokenUsage` is `#[non_exhaustive]`, so a downstream crate cannot
|
||||||
/// write one as a struct expression at all — see `turn_stats`'s note on
|
/// write one as a struct expression at all. Deserializing one is the shortest way to a
|
||||||
/// the same restriction. Deserializing one is the shortest way to a
|
|
||||||
/// populated fixture here, and it keeps the numbers next to the field
|
/// populated fixture here, and it keeps the numbers next to the field
|
||||||
/// names they belong to.
|
/// names they belong to.
|
||||||
///
|
///
|
||||||
|
|
|
||||||
|
|
@ -56,7 +56,7 @@ CREATE UNIQUE INDEX IF NOT EXISTS idx_todos_dedup
|
||||||
ON todos (subsystem, subsystem_key);
|
ON todos (subsystem, subsystem_key);
|
||||||
";
|
";
|
||||||
|
|
||||||
/// Additive column migrations — see `turn_stats.rs`'s identical pattern.
|
/// Additive column migrations.
|
||||||
/// Each runs unconditionally and ignores `duplicate column name` errors, so
|
/// Each runs unconditionally and ignores `duplicate column name` errors, so
|
||||||
/// re-running on an already-migrated db is a silent no-op.
|
/// re-running on an already-migrated db is a silent no-op.
|
||||||
const MIGRATIONS: &[&str] = &[
|
const MIGRATIONS: &[&str] = &[
|
||||||
|
|
|
||||||
|
|
@ -477,8 +477,8 @@ pub async fn drive_turn(
|
||||||
// (badges, stats, auto-reset watermark input).
|
// (badges, stats, auto-reset watermark input).
|
||||||
apply_telemetry(bus, &progress.telemetry);
|
apply_telemetry(bus, &progress.telemetry);
|
||||||
if progress.created {
|
if progress.created {
|
||||||
// Fresh session minted this turn → flag it so the bin loop
|
// Fresh session minted this turn → flag it so the bin loop counts
|
||||||
// mints a `sessions` row + stamps its id onto this turn's stats.
|
// the session boundary in the OTEL metrics.
|
||||||
bus.mark_fresh_session();
|
bus.mark_fresh_session();
|
||||||
bus.emit(LiveEvent::Note {
|
bus.emit(LiveEvent::Note {
|
||||||
text: format!("created fresh session titled \"{}\"", session_title()),
|
text: format!("created fresh session titled \"{}\"", session_title()),
|
||||||
|
|
|
||||||
|
|
@ -1,432 +0,0 @@
|
||||||
//! Per-turn analytics sink. One sqlite row per claude turn captures:
|
|
||||||
//! identity (`model`, `wake_from`, `result_kind`), timing (`started_at`,
|
|
||||||
//! `ended_at`, `duration_ms`), cost (token counts), and behaviour (tool-call
|
|
||||||
//! count + per-tool breakdown).
|
|
||||||
//!
|
|
||||||
//! **Captured but not yet read** (written every turn, no reader in this
|
|
||||||
//! repository — kept for a future chart / backfill): `tool_call_count` (a
|
|
||||||
//! reader recomputes tool totals from `tool_call_breakdown_json` instead),
|
|
||||||
//! `open_threads_count` +
|
|
||||||
//! `open_reminders_count` (planned: a loose-ends-over-time trend), and
|
|
||||||
//! `note` (failure detail for `result_kind = "failed"`).
|
|
||||||
//!
|
|
||||||
//! Lives in the harness dir (`HYPERHIVE_HARNESS_DIR`) as
|
|
||||||
//! `hyperhive-turn-stats.sqlite`; the standalone legacy stats viewer reads it
|
|
||||||
//! and no vacuum sweeps it. Schema is
|
|
||||||
//! intentionally append-only — every column has a default so future
|
|
||||||
//! additions don't break old readers; new columns land via
|
|
||||||
//! `ALTER TABLE ... ADD COLUMN ... DEFAULT ...` in the migration
|
|
||||||
//! block.
|
|
||||||
//!
|
|
||||||
//! Writes are best-effort: a failed insert logs a warning and lets
|
|
||||||
//! the turn loop continue. The next turn either succeeds or the
|
|
||||||
//! operator sees the journal trail.
|
|
||||||
|
|
||||||
use std::path::{Path, PathBuf};
|
|
||||||
use std::sync::Mutex;
|
|
||||||
|
|
||||||
use anyhow::{Context, Result};
|
|
||||||
use rusqlite::{Connection, params};
|
|
||||||
|
|
||||||
/// SQL bootstrap. CREATE TABLE IF NOT EXISTS so first-boot agents
|
|
||||||
/// and existing ones converge on the same shape. The base table is
|
|
||||||
/// fresh-install only; additive migrations land via `MIGRATIONS`
|
|
||||||
/// below as try-and-ignore ALTERs so existing dbs catch up.
|
|
||||||
const SCHEMA: &str = "
|
|
||||||
CREATE TABLE IF NOT EXISTS turn_stats (
|
|
||||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
||||||
started_at INTEGER NOT NULL,
|
|
||||||
ended_at INTEGER NOT NULL,
|
|
||||||
duration_ms INTEGER NOT NULL,
|
|
||||||
model TEXT NOT NULL,
|
|
||||||
wake_from TEXT NOT NULL,
|
|
||||||
input_tokens INTEGER NOT NULL DEFAULT 0,
|
|
||||||
output_tokens INTEGER NOT NULL DEFAULT 0,
|
|
||||||
cache_read_input_tokens INTEGER NOT NULL DEFAULT 0,
|
|
||||||
cache_creation_input_tokens INTEGER NOT NULL DEFAULT 0,
|
|
||||||
last_input_tokens INTEGER NOT NULL DEFAULT 0,
|
|
||||||
last_output_tokens INTEGER NOT NULL DEFAULT 0,
|
|
||||||
last_cache_read_input_tokens INTEGER NOT NULL DEFAULT 0,
|
|
||||||
last_cache_creation_input_tokens INTEGER NOT NULL DEFAULT 0,
|
|
||||||
tool_call_count INTEGER NOT NULL DEFAULT 0,
|
|
||||||
tool_call_breakdown_json TEXT,
|
|
||||||
open_threads_count INTEGER,
|
|
||||||
open_reminders_count INTEGER,
|
|
||||||
result_kind TEXT NOT NULL,
|
|
||||||
note TEXT,
|
|
||||||
session_id INTEGER
|
|
||||||
);
|
|
||||||
CREATE INDEX IF NOT EXISTS idx_turn_stats_started
|
|
||||||
ON turn_stats (started_at DESC);
|
|
||||||
-- NOTE: the index on session_id is created in MIGRATIONS, not here. On an
|
|
||||||
-- existing pre-session db the `CREATE TABLE IF NOT EXISTS` above is a no-op
|
|
||||||
-- (the old table has no session_id column), so indexing session_id in this
|
|
||||||
-- batch would fail with `no such column` and abort the whole SCHEMA apply —
|
|
||||||
-- which disables the stats sink. The column is added by MIGRATIONS first.
|
|
||||||
-- One row per fresh claude session (minted when a session is created rather
|
|
||||||
-- than resumed by title).
|
|
||||||
-- turn_stats.session_id FKs here so per-session stats (first-turn tokens,
|
|
||||||
-- per-session totals, turn count, duration) are one GROUP BY away.
|
|
||||||
CREATE TABLE IF NOT EXISTS sessions (
|
|
||||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
||||||
started_at INTEGER NOT NULL,
|
|
||||||
model TEXT NOT NULL
|
|
||||||
);
|
|
||||||
";
|
|
||||||
|
|
||||||
/// Additive column migrations. Each runs unconditionally and ignores
|
|
||||||
/// `duplicate column name` errors — sqlite < 3.35 lacks
|
|
||||||
/// `ADD COLUMN IF NOT EXISTS`, so try-and-ignore is the portable path.
|
|
||||||
/// New columns MUST carry a default so existing rows decode.
|
|
||||||
const MIGRATIONS: &[&str] = &[
|
|
||||||
"ALTER TABLE turn_stats ADD COLUMN last_input_tokens INTEGER NOT NULL DEFAULT 0",
|
|
||||||
"ALTER TABLE turn_stats ADD COLUMN last_output_tokens INTEGER NOT NULL DEFAULT 0",
|
|
||||||
"ALTER TABLE turn_stats ADD COLUMN last_cache_read_input_tokens INTEGER NOT NULL DEFAULT 0",
|
|
||||||
"ALTER TABLE turn_stats ADD COLUMN last_cache_creation_input_tokens INTEGER NOT NULL DEFAULT 0",
|
|
||||||
// Nullable FK to sessions.id — no default; pre-migration rows stay NULL
|
|
||||||
// (the surface treats NULL as "no session", inert until capture lands).
|
|
||||||
"ALTER TABLE turn_stats ADD COLUMN session_id INTEGER",
|
|
||||||
// Index on session_id — must run AFTER the column is added, so it lives
|
|
||||||
// here rather than in SCHEMA (see the note there). Idempotent.
|
|
||||||
"CREATE INDEX IF NOT EXISTS idx_turn_stats_session ON turn_stats (session_id)",
|
|
||||||
];
|
|
||||||
|
|
||||||
/// One row to be inserted. `Option`-wrapped fields default to NULL
|
|
||||||
/// when the harness couldn't gather them (e.g. socket roundtrip for
|
|
||||||
/// `open_threads` failed) so a partial row beats no row.
|
|
||||||
#[derive(Debug, Clone)]
|
|
||||||
pub struct TurnStatRow {
|
|
||||||
pub started_at: i64,
|
|
||||||
pub ended_at: i64,
|
|
||||||
pub duration_ms: i64,
|
|
||||||
pub model: String,
|
|
||||||
pub wake_from: String,
|
|
||||||
/// Cumulative across every inference in the turn (cost signal).
|
|
||||||
pub input_tokens: u64,
|
|
||||||
pub output_tokens: u64,
|
|
||||||
pub cache_read_input_tokens: u64,
|
|
||||||
pub cache_creation_input_tokens: u64,
|
|
||||||
/// Last inference's usage — the actual context size at turn end.
|
|
||||||
pub last_input_tokens: u64,
|
|
||||||
pub last_output_tokens: u64,
|
|
||||||
pub last_cache_read_input_tokens: u64,
|
|
||||||
pub last_cache_creation_input_tokens: u64,
|
|
||||||
/// Captured, not yet read — the snapshot recomputes tool totals from
|
|
||||||
/// `tool_call_breakdown_json` (see the module doc).
|
|
||||||
pub tool_call_count: u64,
|
|
||||||
/// Per-tool breakdown as JSON: `{"Read":12,"Bash":3,...}`. None
|
|
||||||
/// when no tools were called (saves a sqlite write of `"{}"`).
|
|
||||||
pub tool_call_breakdown_json: Option<String>,
|
|
||||||
/// Post-turn loose-ends snapshot. Captured, not yet read — planned to
|
|
||||||
/// feed a loose-ends-over-time trend.
|
|
||||||
pub open_threads_count: Option<u64>,
|
|
||||||
pub open_reminders_count: Option<u64>,
|
|
||||||
/// `"ok" | "failed" | "prompt_too_long"`.
|
|
||||||
pub result_kind: &'static str,
|
|
||||||
/// Failure detail for `result_kind = "failed"`. Captured, not yet read.
|
|
||||||
pub note: Option<String>,
|
|
||||||
/// FK to `sessions.id` for the fresh claude session this turn belongs
|
|
||||||
/// to. `None` on pre-capture rows (and when the stats db couldn't mint
|
|
||||||
/// a session) so the read side degrades to empty.
|
|
||||||
pub session_id: Option<i64>,
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Thin sqlite wrapper. Cloning is cheap (Arc-shared connection).
|
|
||||||
#[derive(Clone)]
|
|
||||||
pub struct TurnStats {
|
|
||||||
inner: std::sync::Arc<Mutex<Connection>>,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl TurnStats {
|
|
||||||
/// Open the per-agent stats db, creating the file + schema if
|
|
||||||
/// missing. Returns `None` when the db can't be opened (read-only
|
|
||||||
/// fs in tests, missing state dir) — the harness logs and
|
|
||||||
/// continues without a sink rather than failing the turn loop.
|
|
||||||
#[must_use]
|
|
||||||
pub fn open_default() -> Option<Self> {
|
|
||||||
let path = default_path();
|
|
||||||
match Self::open(&path) {
|
|
||||||
Ok(s) => Some(s),
|
|
||||||
Err(e) => {
|
|
||||||
tracing::warn!(
|
|
||||||
error = ?e,
|
|
||||||
path = %path.display(),
|
|
||||||
"turn_stats: open failed; per-turn analytics disabled"
|
|
||||||
);
|
|
||||||
None
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn open(path: &Path) -> Result<Self> {
|
|
||||||
if let Some(parent) = path.parent() {
|
|
||||||
let _ = std::fs::create_dir_all(parent);
|
|
||||||
}
|
|
||||||
let conn = Connection::open(path)
|
|
||||||
.with_context(|| format!("open turn_stats db {}", path.display()))?;
|
|
||||||
// Rollback journal, not WAL: the default, and what every existing
|
|
||||||
// database already uses.
|
|
||||||
conn.busy_timeout(crate::sqlite_writer::BUSY_TIMEOUT)
|
|
||||||
.context("set turn_stats busy_timeout")?;
|
|
||||||
conn.execute_batch(SCHEMA)
|
|
||||||
.context("apply turn_stats schema")?;
|
|
||||||
for stmt in MIGRATIONS {
|
|
||||||
// Ignore "duplicate column name" — the migration already ran.
|
|
||||||
// Any other error is logged but doesn't fail open() because the
|
|
||||||
// base schema works and we'd rather keep the harness alive than
|
|
||||||
// crash on an upgrade hiccup.
|
|
||||||
if let Err(e) = conn.execute(stmt, []) {
|
|
||||||
let msg = e.to_string();
|
|
||||||
if !msg.contains("duplicate column name") {
|
|
||||||
tracing::warn!(error = %msg, stmt, "turn_stats migration failed");
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Ok(Self {
|
|
||||||
inner: std::sync::Arc::new(Mutex::new(conn)),
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Insert a row. Best-effort — logs + swallows errors so a sqlite
|
|
||||||
/// hiccup (locked db, full disk) doesn't crash the harness.
|
|
||||||
///
|
|
||||||
/// # Panics
|
|
||||||
///
|
|
||||||
/// Panics if the internal lock is poisoned.
|
|
||||||
pub fn record(&self, row: &TurnStatRow) {
|
|
||||||
let conn = self.inner.lock().unwrap();
|
|
||||||
let res = conn.execute(
|
|
||||||
"INSERT INTO turn_stats (
|
|
||||||
started_at, ended_at, duration_ms, model, wake_from,
|
|
||||||
input_tokens, output_tokens,
|
|
||||||
cache_read_input_tokens, cache_creation_input_tokens,
|
|
||||||
last_input_tokens, last_output_tokens,
|
|
||||||
last_cache_read_input_tokens, last_cache_creation_input_tokens,
|
|
||||||
tool_call_count, tool_call_breakdown_json,
|
|
||||||
open_threads_count, open_reminders_count,
|
|
||||||
result_kind, note, session_id
|
|
||||||
) VALUES (
|
|
||||||
?1, ?2, ?3, ?4, ?5,
|
|
||||||
?6, ?7,
|
|
||||||
?8, ?9,
|
|
||||||
?10, ?11,
|
|
||||||
?12, ?13,
|
|
||||||
?14, ?15,
|
|
||||||
?16, ?17,
|
|
||||||
?18, ?19, ?20
|
|
||||||
)",
|
|
||||||
params![
|
|
||||||
row.started_at,
|
|
||||||
row.ended_at,
|
|
||||||
row.duration_ms,
|
|
||||||
row.model,
|
|
||||||
row.wake_from,
|
|
||||||
i64::try_from(row.input_tokens).unwrap_or(i64::MAX),
|
|
||||||
i64::try_from(row.output_tokens).unwrap_or(i64::MAX),
|
|
||||||
i64::try_from(row.cache_read_input_tokens).unwrap_or(i64::MAX),
|
|
||||||
i64::try_from(row.cache_creation_input_tokens).unwrap_or(i64::MAX),
|
|
||||||
i64::try_from(row.last_input_tokens).unwrap_or(i64::MAX),
|
|
||||||
i64::try_from(row.last_output_tokens).unwrap_or(i64::MAX),
|
|
||||||
i64::try_from(row.last_cache_read_input_tokens).unwrap_or(i64::MAX),
|
|
||||||
i64::try_from(row.last_cache_creation_input_tokens).unwrap_or(i64::MAX),
|
|
||||||
i64::try_from(row.tool_call_count).unwrap_or(i64::MAX),
|
|
||||||
row.tool_call_breakdown_json,
|
|
||||||
row.open_threads_count
|
|
||||||
.map(|n| i64::try_from(n).unwrap_or(i64::MAX)),
|
|
||||||
row.open_reminders_count
|
|
||||||
.map(|n| i64::try_from(n).unwrap_or(i64::MAX)),
|
|
||||||
row.result_kind,
|
|
||||||
row.note,
|
|
||||||
row.session_id,
|
|
||||||
],
|
|
||||||
);
|
|
||||||
if let Err(e) = res {
|
|
||||||
tracing::warn!(error = ?e, "turn_stats: insert failed");
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Mint a new session row at fresh-session start, returning its `id`
|
|
||||||
/// for stamping onto this session's `turn_stats` rows. Best-effort —
|
|
||||||
/// returns `None` (and logs) on any sqlite error, so a hiccup degrades
|
|
||||||
/// to NULL `session_id` rows rather than crashing the turn loop.
|
|
||||||
///
|
|
||||||
/// # Panics
|
|
||||||
///
|
|
||||||
/// Panics if the internal lock is poisoned.
|
|
||||||
#[must_use]
|
|
||||||
pub fn start_session(&self, started_at: i64, model: &str) -> Option<i64> {
|
|
||||||
let conn = self.inner.lock().unwrap();
|
|
||||||
match conn.execute(
|
|
||||||
"INSERT INTO sessions (started_at, model) VALUES (?1, ?2)",
|
|
||||||
params![started_at, model],
|
|
||||||
) {
|
|
||||||
Ok(_) => Some(conn.last_insert_rowid()),
|
|
||||||
Err(e) => {
|
|
||||||
tracing::warn!(error = ?e, "turn_stats: start_session insert failed");
|
|
||||||
None
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Token counts from the most recently inserted row, if any.
|
|
||||||
/// Returns `(ctx, cost)` — both backfill `Bus` on startup so the
|
|
||||||
/// per-agent web UI's ctx + cost badges paint with real numbers on
|
|
||||||
/// cold load instead of waiting for the next `TokenUsageChanged`
|
|
||||||
/// SSE event. Best-effort: any sqlite error returns `(None, None)`.
|
|
||||||
///
|
|
||||||
/// Pre-migration rows (before the `last_*_tokens` columns existed)
|
|
||||||
/// have last-inference zeros — those rows yield `ctx = None` so the
|
|
||||||
/// badge stays empty until the next real turn rather than showing a
|
|
||||||
/// misleading 0.
|
|
||||||
/// # Panics
|
|
||||||
///
|
|
||||||
/// Panics if the internal lock is poisoned.
|
|
||||||
#[must_use]
|
|
||||||
pub fn last_usage(
|
|
||||||
&self,
|
|
||||||
) -> (
|
|
||||||
Option<hive_claude::TokenUsage>,
|
|
||||||
Option<hive_claude::TokenUsage>,
|
|
||||||
) {
|
|
||||||
let conn = self.inner.lock().unwrap();
|
|
||||||
conn.query_row(
|
|
||||||
"SELECT input_tokens, output_tokens,
|
|
||||||
cache_read_input_tokens, cache_creation_input_tokens,
|
|
||||||
last_input_tokens, last_output_tokens,
|
|
||||||
last_cache_read_input_tokens, last_cache_creation_input_tokens
|
|
||||||
FROM turn_stats
|
|
||||||
-- `id` (AUTOINCREMENT) is monotonic with insertion, so this is
|
|
||||||
-- the most-recently-inserted row even among same-second turns
|
|
||||||
-- (which `started_at DESC` would order arbitrarily).
|
|
||||||
ORDER BY id DESC
|
|
||||||
LIMIT 1",
|
|
||||||
[],
|
|
||||||
|row| {
|
|
||||||
let g = |i: usize| -> rusqlite::Result<u64> {
|
|
||||||
Ok(u64::try_from(row.get::<_, i64>(i)?).unwrap_or(0))
|
|
||||||
};
|
|
||||||
// `TokenUsage` is `#[non_exhaustive]` (hive-claude 0.1.0+) —
|
|
||||||
// struct-expression construction is blocked for downstream
|
|
||||||
// crates entirely, even with a `..base` (functional update
|
|
||||||
// doesn't carve out an exception, per the non_exhaustive
|
|
||||||
// reference semantics). Build off `Default` and assign the
|
|
||||||
// (all-`pub`) fields instead.
|
|
||||||
let mut cost = hive_claude::TokenUsage::default();
|
|
||||||
cost.input_tokens = g(0)?;
|
|
||||||
cost.output_tokens = g(1)?;
|
|
||||||
cost.cache_read_input_tokens = g(2)?;
|
|
||||||
cost.cache_creation_input_tokens = g(3)?;
|
|
||||||
let mut last = hive_claude::TokenUsage::default();
|
|
||||||
last.input_tokens = g(4)?;
|
|
||||||
last.output_tokens = g(5)?;
|
|
||||||
last.cache_read_input_tokens = g(6)?;
|
|
||||||
last.cache_creation_input_tokens = g(7)?;
|
|
||||||
let ctx = if last == hive_claude::TokenUsage::default() {
|
|
||||||
None
|
|
||||||
} else {
|
|
||||||
Some(last)
|
|
||||||
};
|
|
||||||
Ok((ctx, Some(cost)))
|
|
||||||
},
|
|
||||||
)
|
|
||||||
.unwrap_or((None, None))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn default_path() -> PathBuf {
|
|
||||||
crate::paths::harness_dir().join("hyperhive-turn-stats.sqlite")
|
|
||||||
}
|
|
||||||
|
|
||||||
#[cfg(test)]
|
|
||||||
mod tests {
|
|
||||||
use super::*;
|
|
||||||
|
|
||||||
/// A `turn_stats` db in the *pre-session* shape: the original table with no
|
|
||||||
/// `session_id` column, plus the `started_at` index. This is what every
|
|
||||||
/// agent created before the sessions feature has on disk.
|
|
||||||
fn seed_pre_session_db(path: &Path) {
|
|
||||||
let conn = Connection::open(path).unwrap();
|
|
||||||
conn.execute_batch(
|
|
||||||
"CREATE TABLE turn_stats (
|
|
||||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
||||||
started_at INTEGER NOT NULL,
|
|
||||||
ended_at INTEGER NOT NULL,
|
|
||||||
duration_ms INTEGER NOT NULL,
|
|
||||||
model TEXT NOT NULL,
|
|
||||||
wake_from TEXT NOT NULL,
|
|
||||||
input_tokens INTEGER NOT NULL DEFAULT 0,
|
|
||||||
output_tokens INTEGER NOT NULL DEFAULT 0,
|
|
||||||
cache_read_input_tokens INTEGER NOT NULL DEFAULT 0,
|
|
||||||
cache_creation_input_tokens INTEGER NOT NULL DEFAULT 0,
|
|
||||||
last_input_tokens INTEGER NOT NULL DEFAULT 0,
|
|
||||||
last_output_tokens INTEGER NOT NULL DEFAULT 0,
|
|
||||||
last_cache_read_input_tokens INTEGER NOT NULL DEFAULT 0,
|
|
||||||
last_cache_creation_input_tokens INTEGER NOT NULL DEFAULT 0,
|
|
||||||
tool_call_count INTEGER NOT NULL DEFAULT 0,
|
|
||||||
tool_call_breakdown_json TEXT,
|
|
||||||
open_threads_count INTEGER,
|
|
||||||
open_reminders_count INTEGER,
|
|
||||||
result_kind TEXT NOT NULL,
|
|
||||||
note TEXT
|
|
||||||
);
|
|
||||||
CREATE INDEX idx_turn_stats_started ON turn_stats (started_at DESC);",
|
|
||||||
)
|
|
||||||
.unwrap();
|
|
||||||
}
|
|
||||||
|
|
||||||
fn sample_row() -> TurnStatRow {
|
|
||||||
TurnStatRow {
|
|
||||||
started_at: 100,
|
|
||||||
ended_at: 101,
|
|
||||||
duration_ms: 1_000,
|
|
||||||
model: "opus".to_owned(),
|
|
||||||
wake_from: "recv".to_owned(),
|
|
||||||
input_tokens: 10,
|
|
||||||
output_tokens: 5,
|
|
||||||
cache_read_input_tokens: 0,
|
|
||||||
cache_creation_input_tokens: 0,
|
|
||||||
last_input_tokens: 10,
|
|
||||||
last_output_tokens: 5,
|
|
||||||
last_cache_read_input_tokens: 0,
|
|
||||||
last_cache_creation_input_tokens: 0,
|
|
||||||
tool_call_count: 1,
|
|
||||||
tool_call_breakdown_json: None,
|
|
||||||
open_threads_count: None,
|
|
||||||
open_reminders_count: None,
|
|
||||||
result_kind: "ok",
|
|
||||||
note: None,
|
|
||||||
session_id: None,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Regression: a pre-session db must `open()` cleanly (the `session_id`
|
|
||||||
/// index used to live in `SCHEMA` and aborted the apply with
|
|
||||||
/// `no such column`, silently disabling the stats sink), get the column
|
|
||||||
/// added by `MIGRATIONS`, and then accept writes.
|
|
||||||
#[test]
|
|
||||||
fn open_upgrades_pre_session_db_and_writes() {
|
|
||||||
let path = std::env::temp_dir().join("hyperhive-turnstats-pre-session-regression.sqlite");
|
|
||||||
let _ = std::fs::remove_file(&path);
|
|
||||||
seed_pre_session_db(&path);
|
|
||||||
|
|
||||||
let stats = TurnStats::open(&path).expect("open() must succeed on a pre-session db");
|
|
||||||
stats.record(&sample_row());
|
|
||||||
|
|
||||||
let conn = Connection::open(&path).unwrap();
|
|
||||||
let rows: i64 = conn
|
|
||||||
.query_row("SELECT COUNT(*) FROM turn_stats", [], |r| r.get(0))
|
|
||||||
.unwrap();
|
|
||||||
assert_eq!(rows, 1, "the row must be written once the column is added");
|
|
||||||
let has_session_id: i64 = conn
|
|
||||||
.query_row(
|
|
||||||
"SELECT COUNT(*) FROM pragma_table_info('turn_stats') \
|
|
||||||
WHERE name = 'session_id'",
|
|
||||||
[],
|
|
||||||
|r| r.get(0),
|
|
||||||
)
|
|
||||||
.unwrap();
|
|
||||||
assert_eq!(has_session_id, 1, "MIGRATIONS must add session_id");
|
|
||||||
|
|
||||||
let _ = std::fs::remove_file(&path);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -33,4 +33,4 @@ Two bins from one shared lib (`src/lib.rs`):
|
||||||
|
|
||||||
Supporting modules: **`paths.rs`** (daemon socket + agent-socket
|
Supporting modules: **`paths.rs`** (daemon socket + agent-socket
|
||||||
resolution), **`stats.rs`** (the `bash_commands` favorite-tool stat
|
resolution), **`stats.rs`** (the `bash_commands` favorite-tool stat
|
||||||
recorded into turn-stats.sqlite).
|
recorded into hyperhive-turn-stats.sqlite).
|
||||||
|
|
|
||||||
|
|
@ -20,10 +20,9 @@ pub fn tasks_dir() -> PathBuf {
|
||||||
harness_dir().join("bash-tasks")
|
harness_dir().join("bash-tasks")
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Per-agent turn-stats sqlite — the harness writes one row per claude
|
/// Per-agent sqlite in the harness dir: the runner appends normalised
|
||||||
/// turn here; the runner appends normalised bash-command heads to its
|
/// bash-command heads to its `bash_commands` table, which the harness reads for
|
||||||
/// `bash_commands` table, which the harness reads for its OTEL bash-command
|
/// its OTEL bash-command counter.
|
||||||
/// counter.
|
|
||||||
#[must_use]
|
#[must_use]
|
||||||
pub fn turn_stats_db() -> PathBuf {
|
pub fn turn_stats_db() -> PathBuf {
|
||||||
harness_dir().join("hyperhive-turn-stats.sqlite")
|
harness_dir().join("hyperhive-turn-stats.sqlite")
|
||||||
|
|
|
||||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue