Watch
0
0
Fork
You've already forked hyperhive
0

hive-agent: no hard-coded context window sizes, the window is what the API reports

This commit is contained in:
damocles 2026-11-15 20:11:26 +01:00 • committed by mara
commit 1bfee54414
10 changed files with 47 additions and 124 deletions

View file

@ -59,21 +59,13 @@ terminal: a model or effort the swarm sets for the agent replaces the
clearing it goes back to the local value → clearing it goes back to the local value →
[`4-implementation/swarm/inference.md`](../swarm/inference.md). [`4-implementation/swarm/inference.md`](../swarm/inference.md).
`harness_state::context_window_tokens(model)` looks up The context window is whatever the model reports once a turn completes
context-window size per-model. Resolution order (first (`modelUsage.*.contextWindow`, `Bus::api_context_window`). The harness keeps no
match wins): per-model table of window sizes: until the first turn of a process completes the
window is unknown, and what depends on it waits (the compaction watermarks, the
auto-reset, the `compact` tool, the context fill the swarm shows).
1. The harness's per-family defaults, matched when the family name The window drives watermarks; no web endpoint serves it.
is a case-insensitive substring of the active model name:
haiku=200k, sonnet=1M, opus=1M.
2. `HIVE_CONTEXT_WINDOW_TOKENS` — single global override for any
other model (useful in dev / test).
3. Hard fallback: `200_000`.
Once a turn completes, the window the model reports
(`modelUsage.*.contextWindow`) replaces this value.
The effective window drives watermarks; no web endpoint serves it.
**Session identity — a constant title.** Every turn keys on one fixed, **Session identity — a constant title.** Every turn keys on one fixed,
harness-owned session title (`turn::session_title()`, default harness-owned session title (`turn::session_title()`, default
@ -163,8 +155,8 @@ is fleet-wide and set once in `hive-agent/prompts/claude-settings.json`.
The **when** for hyperhive's own policy is a `hive_claude::CompactionPolicy` injected by the harness: The **when** for hyperhive's own policy is a `hive_claude::CompactionPolicy` injected by the harness:
`turn::make_session` builds a `PercentPolicy` that compacts once the `turn::make_session` builds a `PercentPolicy` that compacts once the
model-reported context fill reaches `HIVE_COMPACT_WATERMARK_PERCENT` model-reported context fill reaches `HIVE_COMPACT_WATERMARK_PERCENT`
(default **75%**), falling back to `harness_state::context_window_tokens(model)` (default **75%**) of the window the model reported, so a turn that reports none
for the window on turns the model didn't report one. The swarm can replace never triggers it. The swarm can replace
that percent per agent with its `soft_context_percent` that percent per agent with its `soft_context_percent`
([`inference.md`](../swarm/inference.md)); `turn::SoftLimitPolicy` reads it ([`inference.md`](../swarm/inference.md)); `turn::SoftLimitPolicy` reads it
for each decision. `0` disables proactive for each decision. `0` disables proactive
@ -195,8 +187,8 @@ still applies.
- **Auto session-reset** — a third path (`turn::maybe_auto_reset`, - **Auto session-reset** — a third path (`turn::maybe_auto_reset`,
pre-turn) that fires when both conditions hold: context is ≥ a watermark pre-turn) that fires when both conditions hold: context is ≥ a watermark
(`HIVE_AUTO_RESET_WATERMARK_TOKENS`, default **50% of (`HIVE_AUTO_RESET_WATERMARK_TOKENS`, default **50% of the reported
`context_window_tokens(model)`**) AND the time since the last turn window**, none before one is reported) AND the time since the last turn
exceeds the assumed prompt-cache TTL (`HIVE_CACHE_TTL_SECS`, default exceeds the assumed prompt-cache TTL (`HIVE_CACHE_TTL_SECS`, default
`3600`). Claude's prompt cache goes cold after a while; once it's cold, `3600`). Claude's prompt cache goes cold after a while; once it's cold,
`--resume`-ing a large session pays the full re-upload cost with no `--resume`-ing a large session pays the full re-upload cost with no

View file

@ -105,7 +105,7 @@ hive_name?, swarm_name?, matrix_accounts? }`. `matrix_accounts` is a
- `compact` — queues a compaction of the current session; the only way to - `compact` — queues a compaction of the current session; the only way to
request one. No args. Gated server-side on the last completed request one. No args. Gated server-side on the last completed
turn's context usage: refused (with an explanation, no side effect) turn's context usage: refused (with an explanation, no side effect)
unless usage is above 40% of the effective context window. On a unless usage is above 40% of the reported context window (and refused while no turn has reported one). On a
pass, queues the deferred `compact_pending` flag — consumed at the end pass, queues the deferred `compact_pending` flag — consumed at the end
of the current turn, so it never of the current turn, so it never
races a live claude process, and the usual pre-compaction races a live claude process, and the usual pre-compaction

View file

@ -49,7 +49,8 @@ export interface AgentStateHeader {
model_source: InferenceSource; model_source: InferenceSource;
effort_source: InferenceSource; effort_source: InferenceSource;
resolved_model: string | null; resolved_model: string | null;
context_window_tokens: number; // Null until the agent's first turn reports the window.
context_window_tokens: number | null;
ctx_usage: TokenUsage | null; ctx_usage: TokenUsage | null;
cost_usage: TokenUsage | null; cost_usage: TokenUsage | null;
// Absent from an agent that predates the field. // Absent from an agent that predates the field.

View file

@ -20,8 +20,8 @@ use swarm_wire::agent_inference::Source;
use tokio::sync::broadcast; use tokio::sync::broadcast;
use crate::harness_state::{ use crate::harness_state::{
configured_effort, configured_model, context_window_tokens, local_effort, local_model, configured_effort, configured_model, local_effort, local_model, read_harness_state,
read_harness_state, write_harness_state, write_harness_state,
}; };
const CHANNEL_CAPACITY: usize = 256; const CHANNEL_CAPACITY: usize = 256;
@ -361,11 +361,11 @@ pub struct Bus {
last_turn_ended_unix: Arc<AtomicI64>, last_turn_ended_unix: Arc<AtomicI64>,
/// Per-inference context-window size as reported by the Anthropic API /// Per-inference context-window size as reported by the Anthropic API
/// in the stream-json `result` event (`modelUsage.*.contextWindow`). /// in the stream-json `result` event (`modelUsage.*.contextWindow`).
/// Set by the stdout pump on every completed turn. Takes precedence /// Set by the stdout pump on every completed turn. The only source of the
/// over the per-model defaults in `harness_state::context_window_tokens` /// window: there is no per-model table, so it is `None` until the first
/// for compaction watermark calculations — it reflects the actual /// turn of this process completes, and everything that divides by it
/// limit the model enforces, which may differ from the default /// (the compaction watermarks, the context fill the swarm shows) waits
/// (e.g. 200 k active window on a 1 M cache-enabled model). /// for it.
api_context_window: Arc<Mutex<Option<u64>>>, api_context_window: Arc<Mutex<Option<u64>>>,
/// The swarm's soft context limit, as a percent of the context window /// The swarm's soft context limit, as a percent of the context window
/// (`soft_context_percent` of the agent's `agent-inference` key): the /// (`soft_context_percent` of the agent's `agent-inference` key): the
@ -717,16 +717,6 @@ impl Bus {
*self.soft_context_percent.lock().unwrap() *self.soft_context_percent.lock().unwrap()
} }
/// The effective context window for `model`: the API-reported window if a
/// turn has completed (`api_context_window`), else the per-model default
/// ([`context_window_tokens`]). Single accessor so the state + dashboard
/// endpoints agree by construction.
#[must_use]
pub fn effective_context_window(&self, model: &str) -> u64 {
self.api_context_window()
.unwrap_or_else(|| context_window_tokens(model))
}
/// Walk a stream-json value for `tool_use` blocks and bump the /// Walk a stream-json value for `tool_use` blocks and bump the
/// per-turn counter for each one we find. Called by the stdout /// per-turn counter for each one we find. Called by the stdout
/// pump on every parsed line. Cheap when the line isn't an /// pump on every parsed line. Cheap when the line isn't an

View file

@ -224,47 +224,11 @@ pub fn configured_effort() -> Option<&'static str> {
.map(|s| &*Box::leak(s.into_boxed_str())) .map(|s| &*Box::leak(s.into_boxed_str()))
} }
/// Context-window size in tokens per model family, keyed by a substring
/// of the model name (`"sonnet"` matches `"claude-sonnet-4-5"`).
const CONTEXT_WINDOW_DEFAULTS: [(&str, u64); 3] = [
("haiku", 200_000),
("sonnet", 1_000_000),
("opus", 1_000_000),
];
/// Context-window size in tokens for a given model name, used until a
/// completed turn reports the real window (`Bus::effective_context_window`
/// prefers that one).
///
/// Resolution order (first match wins):
/// 1. [`CONTEXT_WINDOW_DEFAULTS`] — the family key, case-insensitively,
/// is a substring of the model name.
/// 2. `HIVE_CONTEXT_WINDOW_TOKENS` — single global override (any model).
/// 3. Hard fallback: `200_000`.
#[must_use]
pub fn context_window_tokens(model: &str) -> u64 {
let m = model.to_ascii_lowercase();
if let Some((_, tokens)) = CONTEXT_WINDOW_DEFAULTS
.iter()
.find(|(family, _)| m.contains(family))
{
return *tokens;
}
// Global override (single value, any model).
if let Ok(s) = std::env::var("HIVE_CONTEXT_WINDOW_TOKENS")
&& let Ok(v) = s.trim().parse::<u64>()
&& v > 0
{
return v;
}
200_000
}
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::{ use super::{
DEFAULT_EFFORT, DEFAULT_MODEL, Source, context_window_tokens, effort_from, model_from, DEFAULT_EFFORT, DEFAULT_MODEL, Source, effort_from, model_from, read_harness_state,
read_harness_state, write_harness_state, write_harness_state,
}; };
/// A state file an older harness wrote, with the `active_model` nothing /// A state file an older harness wrote, with the `active_model` nothing
@ -310,17 +274,6 @@ mod tests {
assert_eq!(stored["needs_login"], true); assert_eq!(stored["needs_login"], true);
} }
#[test]
fn context_window_defaults_per_model_family() {
assert_eq!(context_window_tokens("claude-haiku-4-5"), 200_000);
assert_eq!(context_window_tokens("haiku"), 200_000);
assert_eq!(context_window_tokens("claude-sonnet-4-5"), 1_000_000);
assert_eq!(context_window_tokens("Claude-Sonnet-4-6"), 1_000_000);
assert_eq!(context_window_tokens("sonnet"), 1_000_000);
assert_eq!(context_window_tokens("claude-opus-4-1"), 1_000_000);
assert_eq!(context_window_tokens("opus"), 1_000_000);
}
#[test] #[test]
fn compiled_default_effort_matches_claude_settings() { fn compiled_default_effort_matches_claude_settings() {
// The value baked into prompts/claude-settings.json. // The value baked into prompts/claude-settings.json.

View file

@ -1064,7 +1064,7 @@ async fn handle_turn<S: Surface>(
let ended_at = chrono::Utc::now().timestamp(); let ended_at = chrono::Utc::now().timestamp();
let duration_ms = i64::try_from(started_instant.elapsed().as_millis()).unwrap_or(i64::MAX); let duration_ms = i64::try_from(started_instant.elapsed().as_millis()).unwrap_or(i64::MAX);
let (open_threads, open_reminders) = S::post_turn_counts(socket).await; let (open_threads, open_reminders) = S::post_turn_counts(socket).await;
let context_window = bus.effective_context_window(&model_at_start); let context_window = bus.api_context_window();
let row = serve_common::build_row(serve_common::TurnRowArgs { let row = serve_common::build_row(serve_common::TurnRowArgs {
started_at, started_at,
ended_at, ended_at,

View file

@ -273,15 +273,18 @@ pub fn record_swarm_queue_publish_skipped(site: &'static str) {
.add(1, &[KeyValue::new("site", site)]); .add(1, &[KeyValue::new("site", site)]);
} }
/// Record the context window's fill at turn end: `used` tokens of `size`, /// Record the context window's fill at turn end: `used` tokens, and the
/// the two numbers the web UI's ctx% divides. No-op when OTEL isn't /// window's `size` once the API has reported one (until then there is nothing
/// configured, same as [`record`]. /// to divide by, so no size is recorded rather than a guessed one). No-op when
pub fn record_context(used: u64, size: u64) { /// OTEL isn't configured, same as [`record`].
pub fn record_context(used: u64, size: Option<u64>) {
let Some(inst) = INSTRUMENTS.get_or_init(build).as_ref() else { let Some(inst) = INSTRUMENTS.get_or_init(build).as_ref() else {
return; return;
}; };
inst.context_used.record(used, &[]); inst.context_used.record(used, &[]);
if let Some(size) = size {
inst.context_size.record(size, &[]); inst.context_size.record(size, &[]);
}
} }
/// Add what an ACP agent reported spending since the last call. No-op when /// Add what an ACP agent reported spending since the last call. No-op when

View file

@ -91,8 +91,9 @@ struct AgentStateMsg {
/// on, resolved from that turn's assistant events. `None` before any /// on, resolved from that turn's assistant events. `None` before any
/// turn has completed. /// turn has completed.
resolved_model: Option<String>, resolved_model: Option<String>,
/// Effective context-window token budget for the current model. /// The context window the API last reported, in tokens; `null` until the
context_window_tokens: u64, /// first turn of this process completes.
context_window_tokens: Option<u64>,
/// Last-inference token usage from the most recent completed turn — the /// Last-inference token usage from the most recent completed turn — the
/// current context-window occupancy. `None` until the first turn ends. /// current context-window occupancy. `None` until the first turn ends.
ctx_usage: Option<hive_claude::TokenUsage>, ctx_usage: Option<hive_claude::TokenUsage>,
@ -178,7 +179,7 @@ fn snapshot(bus: &Bus, login_state: &Mutex<LoginState>) -> AgentStateMsg {
turn_state, turn_state,
turn_state_since: iso8601_utc(since_unix), turn_state_since: iso8601_utc(since_unix),
agent_state: observed_state(), agent_state: observed_state(),
context_window_tokens: bus.effective_context_window(&model), context_window_tokens: bus.api_context_window(),
inference: Effective { inference: Effective {
model, model,
effort: bus.effort(), effort: bus.effort(),
@ -407,7 +408,7 @@ mod tests {
effort_source: Source::AgentNix, effort_source: Source::AgentNix,
}, },
resolved_model: Some("claude-sonnet-4-5-20260805".to_owned()), resolved_model: Some("claude-sonnet-4-5-20260805".to_owned()),
context_window_tokens: 200_000, context_window_tokens: Some(200_000),
ctx_usage: Some(usage(&serde_json::json!({ ctx_usage: Some(usage(&serde_json::json!({
"input_tokens": 11, "input_tokens": 11,
"output_tokens": 22, "output_tokens": 22,

View file

@ -444,13 +444,12 @@ fn compact(bus: &Bus, wake_prompt: Option<String>) -> Response {
message: "compact refused: no completed turn yet — nothing to compact".to_owned(), message: "compact refused: no completed turn yet — nothing to compact".to_owned(),
}; };
}; };
let model = bus.model(); let Some(window) = bus.api_context_window() else {
let window = bus.effective_context_window(&model);
if window == 0 {
return Response::Err { return Response::Err {
message: "compact refused: effective context window is unknown (0)".to_owned(), message: "compact refused: the context window is unknown until a turn reports it"
.to_owned(),
};
}; };
}
// Token counts stay well under 2^53 in practice, so the f64 conversion is exact. // Token counts stay well under 2^53 in practice, so the f64 conversion is exact.
// The ratio is only compared against a fixed threshold, so a last-place rounding error cannot change the outcome. // The ratio is only compared against a fixed threshold, so a last-place rounding error cannot change the outcome.
#[allow(clippy::cast_precision_loss)] #[allow(clippy::cast_precision_loss)]

View file

@ -262,29 +262,14 @@ pub fn stall_sleep_secs() -> u64 {
env_u64_positive("HIVE_STALL_SLEEP_SECS", DEFAULT_STALL_SLEEP_SECS) env_u64_positive("HIVE_STALL_SLEEP_SECS", DEFAULT_STALL_SLEEP_SECS)
} }
/// Resolve the effective context-window size for watermark calculations.
/// Priority order (first wins):
/// 1. API-reported window from the last `result` event's `modelUsage.*.contextWindow`.
/// 2. `harness_state::context_window_tokens` (per-model defaults, then the
/// `HIVE_CONTEXT_WINDOW_TOKENS` override, then 200 000).
///
/// The API-reported window is the authoritative per-inference active
/// context limit. It reflects what the model actually enforces — which
/// for models with large prompt caches (e.g. 1 M total cache) may be
/// significantly smaller than the cache capacity (e.g. 200 k active window
/// for `claude-sonnet-4-6`).
fn effective_context_window(bus: &Bus) -> u64 {
bus.api_context_window()
.unwrap_or_else(|| crate::harness_state::context_window_tokens(&bus.model()))
}
/// Resolve the auto-reset watermark. Priority order: /// Resolve the auto-reset watermark. Priority order:
/// 1. `HIVE_AUTO_RESET_WATERMARK_TOKENS` env var (explicit override). /// 1. `HIVE_AUTO_RESET_WATERMARK_TOKENS` env var (explicit override).
/// 2. 50% of `effective_context_window(bus)`. /// 2. 50% of the window the API last reported, or `0` before one has been.
/// ///
/// `0` disables auto-reset entirely. /// `0` disables auto-reset entirely.
fn auto_reset_watermark_tokens(bus: &Bus) -> u64 { fn auto_reset_watermark_tokens(bus: &Bus) -> u64 {
env_u64("HIVE_AUTO_RESET_WATERMARK_TOKENS").unwrap_or_else(|| effective_context_window(bus) / 2) env_u64("HIVE_AUTO_RESET_WATERMARK_TOKENS")
.unwrap_or_else(|| bus.api_context_window().map_or(0, |window| window / 2))
} }
/// Resolve the assumed cache TTL: `HIVE_CACHE_TTL_SECS` if set, else /// Resolve the assumed cache TTL: `HIVE_CACHE_TTL_SECS` if set, else
@ -353,10 +338,9 @@ impl CompactionPolicy for SoftLimitPolicy {
/// Construct the agent's durable session, with a percent-of-window compaction /// Construct the agent's durable session, with a percent-of-window compaction
/// policy that checkpoints (`CHECKPOINT_PROMPT`) before compacting; on claude /// policy that checkpoints (`CHECKPOINT_PROMPT`) before compacting; on claude
/// also a constant title and the on-disk store. Called once at serve-loop start. /// also a constant title and the on-disk store. Called once at serve-loop start.
/// `percent` comes from a boot-time env var and `default_window` is only a /// `percent` comes from a boot-time env var, so a single build at startup is
/// fallback for turns where the model didn't report a window, so a single /// fine; the window is the one each turn reports (no turn that reports none
/// build at startup is fine; the swarm's soft limit is read from the bus per /// compacts), and the swarm's soft limit is read from the bus per decision.
/// decision.
/// ///
/// # Errors /// # Errors
/// ///
@ -365,7 +349,7 @@ pub fn make_session(bus: &Bus) -> Result<AgentSession> {
let policy = SoftLimitPolicy { let policy = SoftLimitPolicy {
percent: PercentPolicy { percent: PercentPolicy {
percent: compact_percent(), percent: compact_percent(),
default_window: Some(effective_context_window(bus)), default_window: None,
checkpoint_prompt: Some(CHECKPOINT_PROMPT.to_string()), checkpoint_prompt: Some(CHECKPOINT_PROMPT.to_string()),
}, },
bus: bus.clone(), bus: bus.clone(),