vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

fix: self-calibrate context window from API result event

Browse source 
the stream-json result event carries modelUsage.<model>.contextWindow
which is the actual per-inference active window the model enforces.
for claude-sonnet-4-6 this is 200k even though the full prompt cache
can hold millions of tokens via accumulated cache reads.

with the nix-configured sonnet = 1000000 the proactive compact watermark
sat at 750k and was never reached. agents grew context until prompt_too_long
at ~170k — reactive compact, no checkpoint turn.

changes:
- bus gains api_context_window field seeded from modelUsage.*.contextWindow
  in each turn's result event. authoritative; falls back to env var, then 200k.
- new effective_context_window(bus) helper used by both watermark functions
- compact_watermark (75%) and auto_reset_watermark (50%) call effective_context_window
- context_tokens() docstring clarified: all three token fields (input +
  cache_read + cache_creation) count against the per-inference contextWindow
  limit. the large cache_read values seen in the result event are cumulative
  across all inferences in a turn, not per-inference.
- /api/state context_window_tokens now reflects the calibrated window

closes #129
This commit is contained in:
damocles 2026-05-20 22:55:34 +02:00 • committed by Mara
commit b0f6bd8ece
3 changed files with 131 additions and 43 deletions
Download patch file Download diff file Expand all files Collapse all files

102
hive-ag3nt/src/turn.rs
View file

@ -210,11 +210,25 @@ pub fn rate_limit_sleep_secs() -> u64 {
.unwrap_or(DEFAULT_RATE_LIMIT_SLEEP_SECS)
}
/// Resolve the effective context-window size for watermark calculations.
/// Priority order (first wins):
/// 1. API-reported window from the last `result` event's `modelUsage.*.contextWindow`.
/// 2. `HIVE_CONTEXT_WINDOW_TOKENS_*` env vars (Nix-configured per-model defaults).
/// 3. Hard fallback: 200 000.
///
/// The API-reported window is the authoritative per-inference active
/// context limit. It reflects what the model actually enforces — which
/// for models with large prompt caches (e.g. 1 M total cache) may be
/// significantly smaller than the cache capacity (e.g. 200 k active window
/// for `claude-sonnet-4-6`).
fn effective_context_window(bus: &Bus) -> u64 {
bus.api_context_window()
.unwrap_or_else(|| crate::events::context_window_tokens(&bus.model()))
}
/// Resolve the auto-reset watermark. Priority order:
/// 1. `HIVE_AUTO_RESET_WATERMARK_TOKENS` env var (explicit override).
/// 2. 50% of the model's context window (derived from `bus.model()` +
/// `events::context_window_tokens`).
///
/// 2. 50% of `effective_context_window(bus)`.
/// `0` disables auto-reset entirely.
fn auto_reset_watermark_tokens(bus: &Bus) -> u64 {
if let Some(v) = std::env::var("HIVE_AUTO_RESET_WATERMARK_TOKENS")
@ -223,7 +237,7 @@ fn auto_reset_watermark_tokens(bus: &Bus) -> u64 {
{
return v;
}
crate::events::context_window_tokens(&bus.model()) / 2
effective_context_window(bus) / 2
}
/// Resolve the assumed cache TTL: `HIVE_CACHE_TTL_SECS` if set, else
@ -238,9 +252,7 @@ fn cache_ttl_secs() -> u64 {
/// Resolve the proactive-compaction watermark. Priority order:
/// 1. `HIVE_COMPACT_WATERMARK_TOKENS` env var (explicit override).
/// 2. 75% of the model's context window (derived from `bus.model()` +
/// `events::context_window_tokens`).
///
/// 2. 75% of `effective_context_window(bus)`.
/// `0` disables proactive compaction (reactive path still applies).
fn compact_watermark_tokens(bus: &Bus) -> u64 {
if let Some(v) = std::env::var("HIVE_COMPACT_WATERMARK_TOKENS")
@ -249,7 +261,7 @@ fn compact_watermark_tokens(bus: &Bus) -> u64 {
{
return v;
}
crate::events::context_window_tokens(&bus.model()) * 3 / 4
effective_context_window(bus) * 3 / 4
}
/// Drive one turn end-to-end. Three paths layer on top of the raw `run_turn`:
@ -554,39 +566,53 @@ async fn run_claude(prompt: &str, files: &TurnFiles, bus: &Bus) -> Result<(bool,
if line.contains(PROMPT_TOO_LONG_MARKER) {
flag_out.store(true, Ordering::Relaxed);
}
if let Ok(v) = serde_json::from_str::<serde_json::Value>(&line) {
// Rate-limit detection: only fire on JSON `error` events,
// not on arbitrary text content. An agent discussing a past
// rate limit in its response would otherwise trigger a false
// positive (the full conversation flows through stdout as
// stream-json, so any text the model outputs is visible here).
if v.get("type").and_then(|t| t.as_str()) == Some("error") {
let raw = v.to_string();
if RATE_LIMIT_MARKERS.iter().any(|m| raw.contains(m)) {
match serde_json::from_str::<serde_json::Value>(&line) {
Ok(v) => {
// Rate-limit detection: only fire on JSON `error` events,
// not on arbitrary text content. An agent discussing a past
// rate limit in its response would otherwise trigger a false
// positive (the full conversation flows through stdout as
// stream-json, so any text the model outputs is visible here).
if v.get("type").and_then(|t| t.as_str()) == Some("error") {
let raw = v.to_string();
if RATE_LIMIT_MARKERS.iter().any(|m| raw.contains(m)) {
rate_out.store(true, Ordering::Relaxed);
}
}
if let Some(u) = crate::events::TokenUsage::from_assistant_event(&v) {
last_inference = Some(u);
}
if let Some(cost) = crate::events::TokenUsage::from_stream_event(&v) {
// Fallback to `cost` if the turn somehow produced
// a result without any assistant event — keeps the
// ctx badge from going stale on a degenerate turn.
let ctx = last_inference.unwrap_or(cost);
bus_out.record_turn_usage(ctx, cost);
}
// Seed the API-reported context-window from the result
// event's `modelUsage.*.contextWindow` field. This is
// the authoritative per-inference active window used for
// compaction watermarks — it reflects what the model
// actually enforces, which may differ from the Nix
// config (e.g. 200k active window on a 1M cache model).
if let Some(w) =
crate::events::TokenUsage::context_window_from_result_event(&v)
{
bus_out.set_api_context_window(w);
}
bus_out.observe_stream(&v);
bus_out.emit(LiveEvent::Stream(v));
}
Err(_) => {
// Non-JSON stdout: raw text check is fine here since these
// are claude CLI messages, not conversation content.
if RATE_LIMIT_MARKERS.iter().any(|m| line.contains(m)) {
rate_out.store(true, Ordering::Relaxed);
}
bus_out.emit(LiveEvent::Note {
text: format!("(non-json) {line}"),
});
}
if let Some(u) = crate::events::TokenUsage::from_assistant_event(&v) {
last_inference = Some(u);
}
if let Some(cost) = crate::events::TokenUsage::from_stream_event(&v) {
// Fallback to `cost` if the turn somehow produced
// a result without any assistant event — keeps the
// ctx badge from going stale on a degenerate turn.
let ctx = last_inference.unwrap_or(cost);
bus_out.record_turn_usage(ctx, cost);
}
bus_out.observe_stream(&v);
bus_out.emit(LiveEvent::Stream(v));
} else {
// Non-JSON stdout: raw text check is fine here since these
// are claude CLI messages, not conversation content.
if RATE_LIMIT_MARKERS.iter().any(|m| line.contains(m)) {
rate_out.store(true, Ordering::Relaxed);
}
bus_out.emit(LiveEvent::Note {
text: format!("(non-json) {line}"),
});
}
}
});