mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

harness metrics for tool calls, bash commands, reminders and first-turn context, on the agents board

Browse source 
The per-agent /stats page read these from its sqlite; the swarm board had
none of them. hive-agent now counts the tools a turn called, the commands
bash tasks ran (from the bash daemon's own table, newer rows only),
reminders scheduled and delivered, and records the first turn of a fresh
session; the turns tab shows them, with fresh sessions per agent. Claude's
own export has no tool-call counter, so the docs no longer say it does.
This commit is contained in:
iris 2026-11-03 16:05:30 +01:00
commit 23c2527199
2 changed files with 618 additions and 28 deletions
Download patch file Download diff file Expand all files Collapse all files

41
docs/scheduler/observability.md
View file

@ -403,28 +403,33 @@ of metrics per claude turn, recorded the moment the turn ends (not polled).
These are deliberately the fields Claude Code's own built-in export (see
above) can't know about — the harness's own wall-clock timing, what woke the
turn, its own outcome classification, the loose-ends backlog, session
boundaries, and how full the context window is. Token usage and tool-call
counts are **not** duplicated here; that's already covered by Claude's own
export. Cost is here only for agents on an ACP runtime, which have no export
boundaries, how full the context window is, the tools a turn called, the shell
commands bash tasks ran, and reminders. Token usage is **not** duplicated here;
that's already covered by Claude's own export, which has no tool-call counter.
Cost is here only for agents on an ACP runtime, which have no export
of their own — see [ACP-reported cost](#acp-reported-cost).
<!-- vale write-good.Passive = YES -->
| Metric | Unit | Kind | Attributes |
| ---------------------------------------- | --------- | --------- | ------------------------------------------------------------------------------------------------------------------ |
| `hyperhive.agent.turn.duration` | `ms` | histogram | `wake_from`, `result_kind`, `model` |
| `hyperhive.agent.turn.count` | — | counter | `wake_from`, `result_kind`, `model` |
| `hyperhive.agent.session.count` | — | counter | `model` (incremented once per fresh, non-`--continue`'d session) |
| `hyperhive.agent.loose_ends.threads` | — | gauge | none |
| `hyperhive.agent.loose_ends.reminders` | — | gauge | none |
| `hyperhive.agent.todos.open` | — | gauge | none — un-acked todos in the harness-local todo store (reminders excluded), recorded every 30 seconds |
| `hyperhive.agent.claude_md.lines` | — | gauge | none — recorded from the `AGENTS.md`-size watch's own ~15-minute tick, **not** per turn like the rows above |
| `hyperhive.agent.subagent_role.lines` | — | gauge | `role` (the file stem) — one series per role file in `subagent_roles/`, on the same ~15-minute tick |
| `hyperhive.agent.claude_usage.percent` | `%` | gauge | `window` (`five_hour`, `seven_day`, … as the usage endpoint names them) — polled every 5 minutes, **not** per turn |
| `hyperhive.agent.claude_usage.resets_at` | `s` | gauge | `window` — unix seconds at which that window resets; same 5-minute poll |
| `hyperhive.agent.context.used` | `{token}` | gauge | none — tokens in the context window at turn end, the numerator of the web UI's ctx% |
| `hyperhive.agent.context.size` | `{token}` | gauge | none — the context window those tokens fill, ctx%'s denominator |
| `hyperhive.agent.cost.usage` | — | counter | `model`, `currency` — ACP agents only, see below |
| Metric | Unit | Kind | Attributes |
| ---------------------------------------- | --------- | --------- | --------------------------------------------------------------------------------------------------------------------- |
| `hyperhive.agent.turn.duration` | `ms` | histogram | `wake_from`, `result_kind`, `model` |
| `hyperhive.agent.turn.count` | — | counter | `wake_from`, `result_kind`, `model` |
| `hyperhive.agent.session.count` | — | counter | `model` (incremented once per fresh, non-`--continue`'d session) |
| `hyperhive.agent.loose_ends.threads` | — | gauge | none |
| `hyperhive.agent.loose_ends.reminders` | — | gauge | none |
| `hyperhive.agent.todos.open` | — | gauge | none — un-acked todos in the harness-local todo store (reminders excluded), recorded every 30 seconds |
| `hyperhive.agent.claude_md.lines` | — | gauge | none — recorded from the `AGENTS.md`-size watch's own ~15-minute tick, **not** per turn like the rows above |
| `hyperhive.agent.subagent_role.lines` | — | gauge | `role` (the file stem) — one series per role file in `subagent_roles/`, on the same ~15-minute tick |
| `hyperhive.agent.claude_usage.percent` | `%` | gauge | `window` (`five_hour`, `seven_day`, … as the usage endpoint names them) — polled every 5 minutes, **not** per turn |
| `hyperhive.agent.claude_usage.resets_at` | `s` | gauge | `window` — unix seconds at which that window resets; same 5-minute poll |
| `hyperhive.agent.context.used` | `{token}` | gauge | none — tokens in the context window at turn end, the numerator of the web UI's ctx% |
| `hyperhive.agent.context.size` | `{token}` | gauge | none — the context window those tokens fill, ctx%'s denominator |
| `hyperhive.agent.cost.usage` | — | counter | `model`, `currency` — ACP agents only, see below |
| `hyperhive.agent.tool.calls` | — | counter | `tool` — the tool's name, or `Skill:<skill>` for a skill; counted when a turn ends |
| `hyperhive.agent.bash.commands` | — | counter | `command` — the command name a bash task ran (the first real command, past `cd` and `sudo`), counted when a turn ends |
| `hyperhive.agent.reminders` | — | counter | `event` (`scheduled` or `delivered`) |
| `hyperhive.agent.first_turn.context` | `{token}` | gauge | none — input tokens of the first turn of a fresh session, the cold cost of the system prompt and `AGENTS.md` |
An agent with no `subagent_roles/` directory exports no
`subagent_role.lines` series. A role file deleted while the harness runs

2
docs/swarm/services.md
View file

@ -200,7 +200,7 @@ Grafana start — see [`credentials.md`](credentials.md).
`nix/host-modules/swarm-grafana/dashboards/`; edits made in the UI aren't
kept.
- `agents`: per-agent resource usage and Claude Code activity. The **activity** tab ends with a **Busiest agents** table: turns, input, output and cache-read tokens, and cost for each agent in the range, busiest first. The **turns** tab holds what each agent's own `/stats` page shows about turns: average turn duration, cache hit rate and tokens per turn, turns per hour by agent and by model, the turn duration over time (median, 95th percentile and mean), and the mix of wake sources and results. The duration percentiles are right from the first deploy that includes the histogram bounds in `hive-agent`; before it, they read a flat 10 seconds.
- `agents`: per-agent resource usage and Claude Code activity. The **activity** tab ends with a **Busiest agents** table: turns, input, output and cache-read tokens, and cost for each agent in the range, busiest first. The **turns** tab holds what each agent's own `/stats` page shows about turns: average turn duration, cache hit rate and tokens per turn, turns per hour by agent and by model, the turn duration over time (median, 95th percentile and mean), and the mix of wake sources and results. It also has the tools called, the commands bash tasks ran, reminders per hour, the first-turn context size and fresh sessions per agent: everything the `/stats` page shows, except the per-bucket token bars the **cost** tab covers. The duration percentiles are right from the first deploy that includes the histogram bounds in `hive-agent`; before it, they read a flat 10 seconds.
- `attic`: the swarm binary cache. Agents whose latest closure this hive can pull, and why not; the builder's pushes and push failures; closures hive-c0re pulled and deploy steps it refused; the cache's size and store-path count. It reads the closure-pullable gauge, the `swarm-builder` and `hive-c0re` logs, and the `hyperhive.attic.cache_bytes` and `hyperhive.attic.cache_nars` gauges, which a timer in the `swarm-attic` container (`swarm-attic-cache-size`) reads from atticd's database every five minutes and pushes through the container's forwarder. atticd itself exports no metrics and its logs don't reach the log store, so the board has no cache request or hit panel. Filter by hive and agent.
- `authelia`, `bao`, `queue`: the identity provider, the secret store and the NATS queue.
- `claude-usage`: Claude use by model, effort, token type and source. Filter by hive, service (agents or the review classifier) and agent. The service filter separates the classifier's API-key spend from the agents.