alert when claude usage runs ahead of the clock, per account
This commit is contained in:
atlas
mara
parent
dab7768869
commit
319dd6a808
1 changed files with 191 additions and 28 deletions
|
|
@ -411,27 +411,27 @@ of their own — see [ACP-reported cost](#acp-reported-cost).
|
|||
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
| Metric | Unit | Kind | Attributes |
|
||||
| --------------------------------------------- | --------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `hyperhive.agent.turn.duration` | `ms` | histogram | `wake_from`, `result_kind`, `model` |
|
||||
| `hyperhive.agent.turn.count` | — | counter | `wake_from`, `result_kind`, `model` |
|
||||
| `hyperhive.agent.session.count` | — | counter | `model` (incremented once per fresh, non-`--continue`'d session) |
|
||||
| `hyperhive.agent.loose_ends.threads` | — | gauge | none |
|
||||
| `hyperhive.agent.loose_ends.reminders` | — | gauge | none |
|
||||
| `hyperhive.agent.todos.open` | — | gauge | none — un-acked todos in the harness-local todo store (reminders excluded), recorded every 30 seconds |
|
||||
| `hyperhive.agent.claude_md.lines` | — | gauge | none — recorded from the `AGENTS.md`-size watch's own ~15-minute tick, **not** per turn like the rows above |
|
||||
| `hyperhive.agent.subagent_role.lines` | — | gauge | `role` (the file stem) — one series per role file in `subagent_roles/`, on the same ~15-minute tick |
|
||||
| `hyperhive.agent.claude_usage.percent` | `%` | gauge | `window` (`five_hour`, `seven_day`, … as the usage endpoint names them) — polled every 5 minutes, **not** per turn |
|
||||
| `hyperhive.agent.claude_usage.resets_at` | `s` | gauge | `window` — unix seconds at which that window resets; same 5-minute poll |
|
||||
| `hyperhive.agent.context.used` | `{token}` | gauge | none — tokens in the context window at turn end, the numerator of the web UI's ctx% |
|
||||
| `hyperhive.agent.context.size` | `{token}` | gauge | none — the context window those tokens fill, ctx%'s denominator |
|
||||
| `hyperhive.agent.cost.usage` | — | counter | `model`, `currency` — ACP agents only, see below |
|
||||
| `hyperhive.agent.tool.calls` | — | counter | `tool` — the tool's name, or `Skill:<skill>` for a skill; counted when a turn ends |
|
||||
| `hyperhive.agent.bash.commands` | — | counter | `command` — the command name a bash task ran (the first real command, past `cd` and `sudo`), counted when a turn ends |
|
||||
| `hyperhive.agent.reminders` | — | counter | `event` (`scheduled` or `delivered`) |
|
||||
| `hyperhive.agent.first_turn.context` | `{token}` | gauge | none — input tokens of the first turn of a fresh session, the cold cost of the system prompt and `AGENTS.md` |
|
||||
| `hyperhive.agent.swarm_queue.connected` | — | gauge | none — `1` while the swarm-queue client holds a connection, `0` otherwise (including before the first connect), sampled every 30 seconds |
|
||||
| `hyperhive.agent.swarm_queue.publish_skipped` | — | counter | `site` (`terminal` or `agent_state`) — a publish dropped because the client had no connection |
|
||||
| Metric | Unit | Kind | Attributes |
|
||||
| --------------------------------------------- | --------- | --------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `hyperhive.agent.turn.duration` | `ms` | histogram | `wake_from`, `result_kind`, `model` |
|
||||
| `hyperhive.agent.turn.count` | — | counter | `wake_from`, `result_kind`, `model` |
|
||||
| `hyperhive.agent.session.count` | — | counter | `model` (incremented once per fresh, non-`--continue`'d session) |
|
||||
| `hyperhive.agent.loose_ends.threads` | — | gauge | none |
|
||||
| `hyperhive.agent.loose_ends.reminders` | — | gauge | none |
|
||||
| `hyperhive.agent.todos.open` | — | gauge | none — un-acked todos in the harness-local todo store (reminders excluded), recorded every 30 seconds |
|
||||
| `hyperhive.agent.claude_md.lines` | — | gauge | none — recorded from the `AGENTS.md`-size watch's own ~15-minute tick, **not** per turn like the rows above |
|
||||
| `hyperhive.agent.subagent_role.lines` | — | gauge | `role` (the file stem) — one series per role file in `subagent_roles/`, on the same ~15-minute tick |
|
||||
| `hyperhive.agent.claude_usage.percent` | `%` | gauge | `window` (`five_hour`, `seven_day`, … as the usage endpoint names them), `account` (see below) — polled every 5 minutes, **not** per turn |
|
||||
| `hyperhive.agent.claude_usage.resets_at` | `s` | gauge | `window`, `account` — unix seconds at which that window resets; same 5-minute poll |
|
||||
| `hyperhive.agent.context.used` | `{token}` | gauge | none — tokens in the context window at turn end, the numerator of the web UI's ctx% |
|
||||
| `hyperhive.agent.context.size` | `{token}` | gauge | none — the context window those tokens fill, ctx%'s denominator |
|
||||
| `hyperhive.agent.cost.usage` | — | counter | `model`, `currency` — ACP agents only, see below |
|
||||
| `hyperhive.agent.tool.calls` | — | counter | `tool` — the tool's name, or `Skill:<skill>` for a skill; counted when a turn ends |
|
||||
| `hyperhive.agent.bash.commands` | — | counter | `command` — the command name a bash task ran (the first real command, past `cd` and `sudo`), counted when a turn ends |
|
||||
| `hyperhive.agent.reminders` | — | counter | `event` (`scheduled` or `delivered`) |
|
||||
| `hyperhive.agent.first_turn.context` | `{token}` | gauge | none — input tokens of the first turn of a fresh session, the cold cost of the system prompt and `AGENTS.md` |
|
||||
| `hyperhive.agent.swarm_queue.connected` | — | gauge | none — `1` while the swarm-queue client holds a connection, `0` otherwise (including before the first connect), sampled every 30 seconds |
|
||||
| `hyperhive.agent.swarm_queue.publish_skipped` | — | counter | `site` (`terminal` or `agent_state`) — a publish dropped because the client had no connection |
|
||||
|
||||
An agent with no `subagent_roles/` directory exports no
|
||||
`subagent_role.lines` series. A role file deleted while the harness runs
|
||||
|
|
@ -446,6 +446,18 @@ session (API-key backends, not yet logged in) or an expired token skips the
|
|||
poll, so a gauge keeps its last value until the next successful poll — a
|
||||
`resets_at` in the past means the paired `percent` is stale.
|
||||
|
||||
`account` is the `oauthAccount.accountUuid` from the agent's `~/.claude.json`,
|
||||
an opaque id of the claude login that every agent on that login shares. The
|
||||
email address and tokens are never exported. When the file or the field is
|
||||
missing, the series carries no `account` label. Agent builds that predate the label export the
|
||||
series without it.
|
||||
|
||||
The swarm Grafana alert `hh-claude-usage-ahead-of-clock` fires per
|
||||
`(account, window)` when the percent used exceeds the percent of the window
|
||||
that has elapsed, for one hour. It knows `seven_day` (7 days) and `five_hour`
|
||||
(5 hours) and skips other windows and series without `account`. It clears when
|
||||
the clock catches up or the window resets.
|
||||
|
||||
The two `context` gauges hold the two numbers the web UI divides for its ctx%:
|
||||
the last turn's context tokens (input, cache read and cache creation), and the
|
||||
API-reported window, else the model's default. A turn that parsed no usage
|
||||
|
|
|
|||
Loading…
Reference in a new issue