mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

docs: sort the top-level directories into five sections by what the reader wants

Browse source 
Every docs directory moves under 1-getting-started, 2-handbook,
4-implementation or 5-project; the options reference (section 3) comes from
the website and has no directory here. trust-boundary stays whole under the
handbook for now. web-ui, swarm and tools split between handbook and
implementation, and the command-line references move to
2-handbook/reference. Links, path references in code and nix, and the docs
index follow the moves; no page content changes.
This commit is contained in:
iris 2026-11-06 18:30:12 +01:00 • committed by mara
commit 6739bb903d
1 changed files with 736 additions and 701 deletions
Download patch file Download diff file Expand all files Collapse all files

756
docs/4-implementation/scheduler/observability.md Normal file
View file

@ -0,0 +1,756 @@
# Observability (OpenTelemetry)
The swarm has one telemetry pipeline. The swarm's collector writes the
swarm's metrics and log stores and, if you name one, exports upstream; each
hive runs a collector of its own that forwards its agents' telemetry there.
What flows through it: each agent's Claude Code statistics (token usage,
cost, tool call counts) via Claude Code's built-in OpenTelemetry
integration, each agent container's journal, the hive tier's own
`hive-c0re.service` journal line for a detected container crash (see
[Container-crash alerting](#container-crash-alerting-hive-c0re)), and the
hyperhive metrics catalogued below.
## The two collectors
Telemetry crosses two collectors, and which one you configure depends on what
the host is:
| | runs where | receives from | does |
| ------------------------------------ | -------------- | --------------------------------- | ----------------------------------------------- |
| **swarm tier** — `deploy.swarm-otel` | once per swarm | every hive's collector | writes the swarm's stores and exports upstream |
| **hive tier** — always on | every hive | that hive's agents, on the bridge | forwards to the swarm tier. Holds no credential |
An all-local host runs both; the hop between them configures itself. A host that
runs the swarm's services without a hive runs only the swarm tier: the hive tier
labels a hive's telemetry outside its agents' control, and that host has no hive.
The swarm services there push to the swarm collector directly, each as its own
OAuth2 client.
```nix
services.hyperhive.deploy.swarm-otel.upstream = {
endpoint = "https://collector.example.com/otel"; # the upstream
headersCredential = "/run/secrets/otel-headers"; # only the swarm tier reads it
};
```
## Enabling export
Every hive runs its collector and every agent exports to it, with nothing to
switch on: there is no per-hive or per-agent opt-in.
`upstream.endpoint` is where telemetry ends up after it leaves the swarm — optional,
because the swarm's own metrics store (`deploy.victoriametrics`) is a
destination in its own right. With both, telemetry goes to both. See
[`swarm/services.md`](../../2-handbook/swarm/services.md#metrics-victoriametrics--grafana).
**Telemetry leaves a hive exactly one way: through the collector that
runs on the host.** Agents never talk to `upstream.endpoint` themselves —
they export unauthenticated to a collector address only their own containers
can reach. They dial it as
`http://hive-collector.hive:4318`, a name identical on
every hive and every swarm, which each hive's own dnsmasq answers with its bridge IP, so an
agent's build carries no hive-specific address and no swarm domain. The otel module opens that
port on the bridge itself. That collector forwards to the swarm's
([`swarm/services.md`](../../2-handbook/swarm/services.md#telemetry-collector-otel)), the
single process holding the upstream credential and the only writer to the
swarm's stores. No agent holds a copy, and neither does the hive.
The hive collector reaches the swarm collector by its gateway name
(`swarm.otel.domain`, default `otel.<swarm domain>`) — the same DNS-and-CA-trust
shape every hive-to-swarm-service hop uses. On the host running the swarm
collector the hive's dnsmasq answers that name; elsewhere it resolves through
ordinary DNS.
⚠️ **The hive collector carries every bit of that hive's telemetry.** It
runs on the same host as the agents and restarts on failure. Telemetry isn't
the control plane, so degraded telemetry isn't degraded operation.
Both collector tiers queue outgoing batches on disk (`file_storage`) and retry
a failing batch until the receiving end accepts it, so a restart or a short
outage of the next hop delays telemetry instead of dropping it. The queue holds
at most 5000 batches. Two limits remain:
- A full queue makes the exporter refuse new batches and log the drop. The
journald receiver has already moved its cursor past a refused batch, so an
outage longer than the queue can hold still loses data, later and a bounded
amount.
- Retry has no time limit, so the exporter keeps resending a batch that the
store rejects with a retryable error, and enough of those fill the queue.
### Why two tiers
**Agents export only to the hive tier.** Exporting straight to `upstream.endpoint` would mean
every agent needs the credential — and the only place to hand it to an agent
container is somewhere the agent itself can read, its own claude settings
among them. `0600` protects a secret from other containers, not from the
agent it belongs to.
**The tiers stay separate on one box.** An all-local hive is a statement about
_where_ processes run, not about the shape of the deployment. A boundary that
disappears locally is one the local deployment stops testing.
### What the agent→collector hop is and isn't
**It has no application-level auth.** The receiver takes any OTLP that reaches
it; what bounds who can reach it's the firewall — `exposeHostPorts` opens the
port on the bridge interface only — "unauthenticated to a bridge address"
means _reachable from an agent container_, not _presents a credential_.
The consequence, stated because it's a choice rather than an oversight: **any
agent can push arbitrary OTLP, and it's forwarded on under the operator's
credential.** Neither tier can tell a container's genuine Claude Code stats
from anything else shaped like OTLP arriving on that port — including data
smuggled out in resource attributes on an otherwise-legitimate export.
That's a **different risk from the one the collector fixes**: an agent that
held the upstream credential could do everything above _and_ use the token
anywhere else. The collector removes the token and keeps the pipe. Agents are inside the trust
boundary (`docs/2-handbook/trust-boundary/security.md`: capability = accepted risk), so an agent being
able to _send_ is an accepted extension of that boundary — but it's not
closed by this design, and don't read anything here as closing it.
**The `agent` label is self-reported, and the hive→swarm authentication below
doesn't change that.** Treat it as a convenience for grouping dashboards, never
as evidence of which container produced a sample: any agent that can reach this
hive's collector can label its data as any other agent.
<!-- vale write-good.Passive = NO -->
**Logs ride the same hop, and a journal carries more than a counter does.** Each
agent container forwards its own journal through this port — every unit in it at
`info` and above, not an allowlist. That's the harness, the MCP daemons and
whatever a tool call spawned, so command lines and error text leave the
container, not just counts. Destination, credential and trust boundary are the
same as for any OTLP the agent sends.
<!-- vale write-good.Passive = YES -->
Worth spelling out, because two different hops are in play and only one of them
carries a credential:
- **agent→collector** (this section's hop) stays open on the bridge. Nothing
downstream can tell one agent's export from another's.
- **hive→swarm** authenticates. The swarm tier gives each hive its own receiver
and stamps `hive=` from whichever one accepted the sample, so _that_ label is
unforgeable — see [Authenticated ingest](#authenticated-ingest) below.
A verified `hive` is reachable and a verified `agent` isn't — and that falls
out of the topology rather than being a gap someone forgot to close. The swarm
runs one collector, and the mechanism gives it no finer grain: a bearer-token
check never reveals _which_ token matched, and a receiver reads request metadata
rather than the claims it authenticated with.
If you need per-agent numbers you can act on, take them from the agent's own
turn-stats rather than from a metric label.
### Authenticated ingest
The swarm tier gives **each hive its own receiver**, and stamps the `hive` label
from whichever receiver accepted a sample. A hive therefore can't report
metrics as another hive, and can't relabel its own by editing what it sends —
the label isn't taken from the payload at all.
**Nothing to set where the hive holds its store identity.** The hive's collector
reads its secret from the swarm's secret store under the hive's own leaf
(`deploy.bao.clientCertFile`): `swarm-bao-hive-otel-oidc.service` fetches
`swarm/hives/<hive>/oidc/client`, which authelia's mint, in its container,
writes, into `/run/hive-otel-oidc/` (tmpfs, so never on persistent disk). It
runs at boot and retries every 30s until the store answers; until then the
collector runs on an empty file and the swarm refuses its exports. When a
run lands a changed value it restarts the collector. A timer runs it again
every ten minutes, so a secret rotated later reaches the collector within
about ten minutes. On a hive that
runs the store the leaf is already there; elsewhere it's the credential you
place — see [secret placement](../../2-handbook/swarm/credentials.md).
To deliver the secret some other way, name the file. The fetch unit isn't
rendered, and the collector waits up to 120s for the file to appear:
```nix
services.hyperhive.otel.clientSecretFile = "/run/secrets/hive-telemetry.secret";
```
Setting the option to `/var/lib/hive-otel-oidc/hive-<name>.secret`, the path
the fetch used to write, now counts as delivering the file yourself: nothing
refreshes it, and the build warns. Remove the setting to fetch from the store.
**No unauthenticated mode exists.** A hive always presents an identity, so a
missing credential is a build error rather than a quieter fallback — the
collector has no anonymous route to accept samples on, and every path it serves
belongs to exactly one hive.
Getting the secret wrong shows up as the hive's collector logging 401s from the
swarm tier and no metrics appearing for that hive.
⚠️ **`upstream.endpoint` must be valid for `upstream.protocol`.** The upstream exporter follows
`upstream.protocol` (`grpc` → the gRPC exporter, otherwise OTLP/HTTP), and the gRPC
exporter takes an _address_: `https://host/path` is a legal
`OTEL_EXPORTER_OTLP_ENDPOINT` for HTTP but fails as gRPC with _"missing port in
address"_. nix validates the collector's config at build time, so a mismatch is
a build error naming the reason rather than telemetry silently going nowhere.
## Network access
Agent containers reach the host only on ports 80 and 443 (plus DNS). The
collector port needs nothing from you: the hive collector opens it on the bridge.
To let agents reach some other host-local service of your own →
[`networking/network.md`](../networking/network.md#reaching-host-services-exposehostports).
## Options reference
The nix modules (`nix/host-modules/otel.nix`, and `nix/host-modules/swarm-otel.nix`
for the `services.hyperhive.deploy.swarm-otel.upstream.*` export) generate every
`services.hyperhive.otel.*` option's full type, default, description and
example into the [options reference](https://hyperhive.darkest.space/options/)
(host options — `nix build .#docs-host` for a local render). The same module
fixes the collector port every hive and its agents share,
`services.hyperhive.swarm.hiveOtel.collectorPort`, and each hive's own
`services.hyperhive.deploy.hive-controller.otel.extraResourceAttributes`.
That's the reference; this page covers what a per-option listing can't: the
two-tier architecture, the security model, and how the options interact.
## Built-in resource labels
The harness sets the OTLP variables (`OTEL_EXPORTER_OTLP_ENDPOINT`, `_PROTOCOL`,
`OTEL_RESOURCE_ATTRIBUTES`, the temporality preference) **container
wide**, for every unit and in `/etc/profile`, so every
process in an agent container exports to the hive's collector without any
per-tool wiring. That covers Claude Code, `hive-metric`, and anything you run
yourself from a tool call or a shell you open in the container.
Every agent's export therefore includes these resource attributes
automatically:
| Attribute | Value |
| -------------- | ------------------------------------------------------------ |
| `service.name` | `hyperhive-agent` (constant) |
| `agent` | agent logical name (for example `iris`) |
| `swarm` | swarm display name (`services.hyperhive.swarm.name`, if set) |
The swarm's collector adds `hive`, the name of the hive whose receiver accepted
the push (see [Authenticated ingest](#authenticated-ingest)). The container
doesn't set it.
Append additional labels for one hive via `extraResourceAttributes` (see
option reference above). Agents get them at container start, so a change
reaches an agent at its next start, not through a rebuild. Pass custom
per-data-point labels with `hive-metric --labels` (see below).
## Log severity
Every journald receiver in the pipeline maps the journal's `PRIORITY` onto an
OpenTelemetry severity, so a line arrives in the log store already carrying its
level and `severity_text:ERROR` is a query you can write:
| `PRIORITY` | syslog | severity |
| ---------- | ------------------ | -------- |
| 0, 1, 2 | emerg, alert, crit | `FATAL` |
| 3 | err | `ERROR` |
| 4 | warning | `WARN` |
| 5, 6 | notice, info | `INFO` |
| 7 | debug | `DEBUG` |
Both tiers that read a journal do this — the agent container's forwarder over
its own journal, and the swarm collector over the host's — and both import the
same mapping from `nix/journald-severity.nix`, which holds the table once.
Note the direction. Syslog's priority counts **down** in urgency, where
OpenTelemetry's severity number counts **up**. Neither is a passthrough of the
other, and a pipeline that hands one straight to the other renders debug lines
as critical without erroring anywhere.
A record that reaches the store from somewhere other than a journal has no
priority to map. Claude Code's own telemetry is the live example: it exports
OTLP log records directly (`OTEL_LOGS_EXPORTER`, see
`nix/agent-modules/claude-settings.nix`) and sets no severity on them, so they
store as `Unspecified`. The **Log rows with no severity** panel on the
`hyperhive · logs` dashboard counts both cases and keeps them apart: lines that
had a priority and arrived without a severity anyway signal a broken mapping,
while lines that never had one signal nothing of the sort.
### Why Explore's level buttons need the datasource told
The stored field is `severity_text`, and Grafana's log-level buttons filter on
a field called `level` — a name no row here carries. That's not something the
pipeline can fix: VictoriaLogs' OTLP ingester names the stored field
`severity_text` unconditionally (v1.52.0, `app/vlinsert/opentelemetry/pb.go`),
and its ingest parameters have no `_level_field` sibling to rename it with.
`nix/host-modules/swarm-grafana.nix` tells the datasource instead,
provisioning it with a `logLevelRules` entry per severity in the table
above, each matching `severity_text` against one of
those values — which is the plugin's way of saying "the level lives in this
field." `logLevelRules` is the datasource plugin's own jsonData key and the
only level-related one it has — there is no single "the level lives in field
X" string to set, so mapping every level takes one rule each. A level button
then filters on `level` **or** the matching `severity_text`, and the store
answers the second half. Each enabled rule appends an `OR
<field>:<op>"<value>"` term to the query a level button emits, beside the
`level:…` term that matches nothing here. Selecting "info" goes from
```
level:contains_common_case("info","information","informational","notice")
```
to that OR `severity_text:="INFO"`, which the rows here do match.
`Unspecified` is deliberately left out of those rules. It's VictoriaLogs' own
rendering of an absent severity, and it's what the **Log rows with no
severity** panel counts — giving it a level would dress the missing data up
as a colour and retire the instrument that measures it.
#### Three ways to get a `logLevelRules` entry wrong, all of them silent
Grafana accepts any jsonData it doesn't recognise, so a bad rule provisions
cleanly and the affected level button goes on returning zero rows:
- `enabled` must be literally `true`, not merely not-false. The query
builder keeps rules on `rule.enabled` being truthy while the row colouring
path keeps them on `!== false`, so an omitted flag colours rows correctly
and leaves the buttons broken — working in the half nobody is looking at.
- `level` must be a canonical Grafana `LogLevel` value. The builder groups
rules by it and only ever looks up `critical`, `error`, `warning`, `info`,
`debug`, `trace`; `warn` and `fatal` are enum aliases that resolve to
other spellings and match no group.
- the builder compares `value` with `===`, so it must be the severity text
exactly as stored: the uppercase OpenTelemetry short names `overwrite_text`
writes in `nix/journald-severity.nix`, not that table's lowercase keys.
## Host-emitted container-resource metrics (hive-c0re)
<!-- vale write-good.Passive = NO -->
**hive-c0re itself** also exports each agent
container's resource load — the same cgroup gauges —
to this hive's own collector, exactly like an agent does and with
no separate toggle. These come from the host, not the in-container Claude SDK,
so they cover containers even when their agent is idle.
<!-- vale write-good.Passive = YES -->
Emitted via the OpenTelemetry Rust SDK, using the
[semconv `container.*`](https://opentelemetry.io/docs/specs/semconv/system/container-metrics/)
metric names + the standard `container.name` attribute where a spec metric
exists, so off-the-shelf OTEL/Grafana container dashboards work. Resource
`service.name = hyperhive-c0re`; hive-c0re tags each data point `container.name`
(= the `h-<agent>` machine) and the hive `agent` label:
| Metric | Unit | Kind | Source |
| --------------------------------------- | ---- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `container.cpu.time` | `s` | counter | cumulative `cpu.stat` `usage_usec` → seconds |
| `container.memory.usage` | `By` | gauge | `memory.current` |
| `hyperhive.container.memory.limit` | `By` | gauge | `memory.max` (custom — semconv has no `.limit` metric; omitted when unlimited) |
| `hyperhive.container.memory.peak` | `By` | gauge | `memory.peak` (custom — no semconv metric; omitted if unavailable) |
| `hyperhive.container.storage.usage` | `By` | gauge | `state/` + `work/` + the root template (the live ephemeral layer isn't counted), measured by `hive-priv` every 30 min (custom — semconv only has `disk.io`; omitted while unknown: not measured yet, or the last measurement failed) |
| `hyperhive.container.cpu.percent` | `%` | gauge | host-normalised percent (custom — no `rate()` needed) |
| `hyperhive.agent.unrestricted_internet` | `1` | gauge | the agent's `unrestricted_internet` flag in the swarm permissions this hive last read: `1` on, `0` off; a flag never set reads `1`. Omitted while the stored permissions file is unreadable |
| `hyperhive.agent.closure_pullable` | `1` | gauge | `1` when this hive can fetch the agent's latest closure from the swarm's binary cache and verify its signature, else `0` with a `reason` label (`no-record`, `not-cached`, `no-credentials`, `fetch-failed`, `bad-signature`; see [the builder docs](../swarm/builder.md#what-hives-do-with-the-cache)). Omitted before the first probe pass, every 5 minutes |
The `hyperhive.`-prefixed metrics have no semconv equivalent (memory
limit + peak, on-disk footprint, and an instantaneous cpu percent kept
alongside the spec `container.cpu.time` counter for convenience). The `swarm`
label rides on the resource, and the swarm collector adds `hive` from the
receiver that accepted the sample. Any other label goes in the hive's
`extraResourceAttributes`.
<!-- vale write-good.Passive = NO -->
**An absent `hyperhive.agent.unrestricted_internet` series doesn't mean off.**
A hive that predates the metric exports no series at all, and neither does a
hive that can't read its stored permissions. Only a `0` means the flag is off.
Count the agents still on with
`sum({"hyperhive.agent.unrestricted_internet"} == 1)`; dotted names need the
quoted-name selector.
The export cadence is a fixed 60s. Transport is OTLP/HTTP
(JSON) to the hive collector's bridge address, with no auth header — that
first hop is unauthenticated for every producer on this host, and the upstream
credential stays on the swarm tier.
## Container-crash alerting (hive-c0re)
The crash watcher (`crash_watch.rs`, a 10s poll of every managed container)
classifies a container that stopped without an operator-initiated action as
a crash and `tracing::warn!`s `"container crash detected"` — nothing else:
no broker event, no per-hive dashboard banner. Both of those used to exist;
this change removed them in favour of the log line.
The hive tier's own collector runs a `journald` receiver scoped to the units
in `services.hyperhive.otel.journalUnits`: `hive-c0re.service`, the `container@*.service`
units (the start-time rootfs check lines), plus
`squid.service` and `squid-selfcheck.service` (see [Egress proxy](../networking/network.md#egress-proxy-squid)).
A host that runs the cache without the swarm collector adds `swarm-bao-attic-secret.service`
and `swarm-bao-attic-tokens.service`.
A host that also runs the swarm collector names none of these: its whole-host journal receiver
ships them already, and the hive collector then has no `journald` receiver.
No bind mount is needed. Unlike an agent's journal, this collector already
shares a filesystem with the units it reads, because it's a host systemd
service rather than a container. It forwards the crash log line alongside the
rest of the hive's telemetry. A
swarm-level grafana alert, `hh-container-crashed` in
[`nix/host-modules/swarm-grafana.nix`](../../../nix/host-modules/swarm-grafana.nix),
matches it directly and pages the operator — no agent, and no per-hive
dashboard, in the loop.
Requires `services.journald.storage = "persistent"` (the NixOS default) on
every hive host — asserted at eval time.
## Agent-emitted per-turn metrics (`hive-agent`)
<!-- vale write-good.Passive = NO -->
The harness itself (`hive-agent`) exports one small set
of metrics per claude turn, recorded the moment the turn ends (not polled).
These are deliberately the fields Claude Code's own built-in export (see
above) can't know about — the harness's own wall-clock timing, what woke the
turn, its own outcome classification, the loose-ends backlog, session
boundaries, how full the context window is, the tools a turn called, the shell
commands bash tasks ran, and reminders. Token usage is **not** duplicated here;
that's already covered by Claude's own export, which has no tool-call counter.
Cost is here only for agents on an ACP runtime, which have no export
of their own — see [ACP-reported cost](#acp-reported-cost).
<!-- vale write-good.Passive = YES -->
| Metric | Unit | Kind | Attributes |
| ---------------------------------------- | --------- | --------- | --------------------------------------------------------------------------------------------------------------------- |
| `hyperhive.agent.turn.duration` | `ms` | histogram | `wake_from`, `result_kind`, `model` |
| `hyperhive.agent.turn.count` | — | counter | `wake_from`, `result_kind`, `model` |
| `hyperhive.agent.session.count` | — | counter | `model` (incremented once per fresh, non-`--continue`'d session) |
| `hyperhive.agent.loose_ends.threads` | — | gauge | none |
| `hyperhive.agent.loose_ends.reminders` | — | gauge | none |
| `hyperhive.agent.todos.open` | — | gauge | none — un-acked todos in the harness-local todo store (reminders excluded), recorded every 30 seconds |
| `hyperhive.agent.claude_md.lines` | — | gauge | none — recorded from the `AGENTS.md`-size watch's own ~15-minute tick, **not** per turn like the rows above |
| `hyperhive.agent.subagent_role.lines` | — | gauge | `role` (the file stem) — one series per role file in `subagent_roles/`, on the same ~15-minute tick |
| `hyperhive.agent.claude_usage.percent` | `%` | gauge | `window` (`five_hour`, `seven_day`, … as the usage endpoint names them) — polled every 5 minutes, **not** per turn |
| `hyperhive.agent.claude_usage.resets_at` | `s` | gauge | `window` — unix seconds at which that window resets; same 5-minute poll |
| `hyperhive.agent.context.used` | `{token}` | gauge | none — tokens in the context window at turn end, the numerator of the web UI's ctx% |
| `hyperhive.agent.context.size` | `{token}` | gauge | none — the context window those tokens fill, ctx%'s denominator |
| `hyperhive.agent.cost.usage` | — | counter | `model`, `currency` — ACP agents only, see below |
| `hyperhive.agent.tool.calls` | — | counter | `tool` — the tool's name, or `Skill:<skill>` for a skill; counted when a turn ends |
| `hyperhive.agent.bash.commands` | — | counter | `command` — the command name a bash task ran (the first real command, past `cd` and `sudo`), counted when a turn ends |
| `hyperhive.agent.reminders` | — | counter | `event` (`scheduled` or `delivered`) |
| `hyperhive.agent.first_turn.context` | `{token}` | gauge | none — input tokens of the first turn of a fresh session, the cold cost of the system prompt and `AGENTS.md` |
An agent with no `subagent_roles/` directory exports no
`subagent_role.lines` series. A role file deleted while the harness runs
keeps its last value until the harness restarts.
For the two `claude_usage` gauges the harness polls the Claude subscription
usage endpoint (`GET /api/oauth/usage`, the one behind claude's own `/usage`)
with the OAuth access token from the agent's own
`~/.claude/.credentials.json`. The harness only reads that token and never
refreshes it — claude owns refresh-token rotation. An agent with no OAuth
session (API-key backends, not yet logged in) or an expired token skips the
poll, so a gauge keeps its last value until the next successful poll — a
`resets_at` in the past means the paired `percent` is stale.
The two `context` gauges hold the two numbers the web UI divides for its ctx%:
the last turn's context tokens (input, cache read and cache creation), and the
API-reported window, else the model's default. A turn that parsed no usage
leaves both where the previous turn put them. The dashboard's **Context
window used by agent** panel (health tab) divides one by the other.
Resource attributes (`service.name`, `agent`, `swarm`) come from the
same container-wide `OTEL_RESOURCE_ATTRIBUTES` as everything else in this
section, and the swarm's collector adds `hive` — nothing extra to configure.
The export cadence is a fixed 60s, the same
as `hive-c0re`'s container-resource export above. That only controls how often
the harness flushes the batched points to the collector, not how often it
records them (every turn, always).
### ACP-reported cost
An ACP agent reports what its session has cost so far in the `cost` field of
its `usage_update` notifications — a running total, not a per-turn figure
(opencode sums every assistant message in the session). The harness counts
what each report adds to the last one and exports that as
`hyperhive.agent.cost.usage`, so `increase()` over it gives the money spent
in a range. It follows a few rules:
- A new session's first report counts in full, since the session started at
zero.
- A session loaded into a freshly started agent (after a harness restart or
an agent crash) has an unknown total until it reports one, so that first
report only sets the baseline — the turn it covers goes uncounted.
- A total lower than the previous one means the agent dropped part of the
session's history; it adds nothing and becomes the new baseline.
- Cost from a compaction the harness runs between turns counts toward the
next turn.
The value is in whatever `currency` the agent names; opencode always sends
`USD`, and the `hyperhive · agents` dashboard's ACP cost panels (cost tab)
filter on it. A claude agent never records this metric: claude's own export
already carries its cost as `claude_code.cost.usage`, and keeping the two
apart means no turn is ever counted twice. Add both names together for a
whole-swarm spend figure.
## Hive-scoped metrics (hive-c0re)
<!-- vale write-good.Passive = NO -->
hive-c0re measures everything above **per agent**, tagged with the hive it
runs in. It measures these five per **hive** instead, and they carry no
`agent` label — so a hive
that hosts no agents still reports, and "this hive is quiet" is
distinguishable from "this hive is gone." Select them with
`{hive!="",agent=""}`.
<!-- vale write-good.Passive = YES -->
| Metric | Unit | Kind | Meaning |
| -------------------------------------- | ---- | ----- | -------------------------------------------------------------------------------------------------------------------- |
| `process.uptime` | `s` | gauge | seconds since this hive's `hive-c0re` started exporting; a restart reads as a drop to ~0 |
| `hyperhive.hive.degraded` | `1` | gauge | `1` while the hive reports itself unhealthy — the same verdict `/health/ready` gives and the swarm status view shows |
| `hyperhive.hive.warnings` | `1` | gauge | how many warnings are currently raised, split by a `level` attribute (`warn`, `crit`) |
| `hyperhive.hive.warning` | `1` | gauge | `1` per raised warning, with `kind` and `level` attributes; a cleared warning stops reporting |
| `hyperhive.hive.nix_store.utilization` | `1` | gauge | used fraction (0–1) of the filesystem that holds the nix store; no sample when the probe fails |
hive-c0re reports both levels of `hyperhive.hive.warnings` every cycle, `0`
included, so a healthy hive is visible as zeros rather than as missing
series.
`hyperhive.hive.degraded` is what a dashboard should alert on: it's
`hive-c0re`'s own readiness verdict, so it stays in step with `/health/ready`
and with what the swarm controller sees. `hyperhive.hive.warnings` is the
detail behind it — `warn`-level entries mean "an operator should look" and do
**not** set `degraded`. `hyperhive.hive.warning` names the cause: the `kind`
label is one of a fixed set in the source (for example `disk_pressure`, a
failing sweep such as `swarm_status_publish`, `pending_logins`), and the message lives in the
`hive warning raised` line hive-c0re logs when it raises the warning.
The swarm Grafana (`nix/host-modules/swarm-grafana.nix`) alerts on these:
| Alert | Fires when |
| ----------------------- | ----------------------------------------------------------------------------------- |
| `hh-hive-degraded` | `hyperhive.hive.degraded` is `1` for 2 minutes |
| `hh-hive-warning` | any `hyperhive.hive.warning` series other than `disk_pressure` exists for 2 minutes |
| `hh-host-disk-high` | `hyperhive.hive.nix_store.utilization` stays above 0.85 for 10 minutes |
| `hh-host-disk-critical` | `hyperhive.hive.nix_store.utilization` stays above 0.95 for 2 minutes |
The disk alerts replace the old dashboard banner for the nix store, and
`hh-hive-warning` replaces it for every other warning. Disk pressure has its
own pair so the thresholds sit on the metric rather than on the registry
entry.
Same cadence, transport and resource labels as the container metrics above.
## VCS activity metrics (`swarm-controller`)
`swarm-controller` registers a single instance-wide Forgejo webhook (a
"global/system" hook, not scoped to any one org or repo) and counts commit
and push activity as deliveries arrive — occurrence-driven, not polled.
Forgejo's own native `/metrics` endpoint has no equivalent: it exposes
counts of durable rows (issues, comments, repos), and Forgejo doesn't
store either a commit or a push anywhere as a row to count.
| Metric | Unit | Kind | Attributes |
| ---------------------------- | ---- | ------- | ------------------- |
| `hyperhive.vcs.commit.count` | — | counter | `repo` (`org/repo`) |
| `hyperhive.vcs.push.count` | — | counter | `repo` (`org/repo`) |
A push with zero commits (a branch delete, or a force-push that doesn't add
new commits) still increments `push.count`; `commit.count` only advances
when the delivery actually carries commits. Same enable signal (`OTEL_EXPORTER_OTLP_ENDPOINT`), fixed 60s cadence
and `HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES`
resource-attribute channel as `swarm-controller`'s other OTEL exporter (its
`hive-jobq-metrics`-backed job-graph rollup, undocumented here — see that
crate's own doc comment) — `swarm-controller` sets `service.name =
swarm-controller` directly rather than reading it from the container
environment, since it's a standalone daemon, not a per-agent harness process.
Both exporters push through `swarm-controller`'s own authenticated client,
and the collector's `oidc/swarm` receiver checks the bearer. When a push
fails, `swarm-controller` logs `otel export: push to the collector failed`
at warn, with the kind of failure (`token`, `transport` or `status <code>`)
and the start of the response body. It repeats a persisting failure every
10 minutes with a count, and logs once at info when pushes succeed again.
## Review classifier result metrics (`swarm-controller`)
`swarm-controller` counts what review requests and review runs come to, from
the same exporter as the VCS metrics above. The classifier service adds the
retry counter beside its cost counters
([`review-classifier.md`](../swarm/review-classifier.md#observability-and-cost)).
| Metric | Unit | Kind | Attributes |
| ------------------------------------------ | ---- | --------- | ------------------------------------------------------------------------------------ |
| `hyperhive.review_classifier.requests` | — | counter | `outcome`: `queued`, `queue_failed`, `unparsable`, `deferred`, `started`, `reposted` |
| `hyperhive.review_classifier.verdicts` | — | counter | `outcome`: `approve`, `request_changes`, `no_verdict` |
| `hyperhive.review_classifier.checks` | — | counter | `check` (the rule's name), `result`: `match`, `no_match`, `unsure`, `no_answer` |
| `hyperhive.review_classifier.run.duration` | `s` | histogram | `outcome`: the verdict names, or `post_failed` |
| `hyperhive.review_classifier.ask.retries` | — | counter | `runtime`: `api_key`, `acp` |
A verdict means the classifier decided, before it posts the review: a run
whose post fails still adds to `verdicts`, and shows as `post_failed` in
`run.duration`. A repo with no rules file gets an approval with no model call and
counts as an `approve` verdict and run, with no `checks`.
`deferred` counts each time the queue hands over a request for a PR whose
review is still going, so a long review adds one per retry wait. `reposted`
is a request for a head already reviewed: it posts the earlier verdict and
adds to `verdicts` but runs no checks, so it adds nothing to `checks` or
`run.duration`. A request the queue refuses posts a comment and counts
as `queue_failed` and a `no_verdict`. `no_answer` is a classification that
failed, so a `no_verdict` run with a `no_answer` check is the usual cause of
a comment instead of a review. The `check` values are the names in a repo's
rules file, at most 10 per repo.
The `review-classifier` dashboard in Grafana charts these.
## Swarm builder metrics
`swarm-builder` exports what it builds through the OTLP receiver of its
container's own forwarder, which pushes to the swarm collector as the builder's
client. `nix/host-modules/swarm-builder.nix` sets
`OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_RESOURCE_ATTRIBUTES` for the daemon. The exporter is the same setup as the review
classifier's: HTTP/JSON, cumulative, one export a minute, and nothing exported
without `OTEL_EXPORTER_OTLP_ENDPOINT`.
| Metric | Unit | Kind | Attributes |
| ------------------------------------------ | ---- | --------- | ---------------------------------------------------------------- |
| `hyperhive.swarm_builder.builds` | — | counter | `kind`: `pr`, `meta`; `outcome`: `success`, `failure`, `refused` |
| `hyperhive.swarm_builder.build.duration` | `s` | histogram | `kind`, `outcome`: `success`, `failure` |
| `hyperhive.swarm_builder.queue.depth` | — | gauge | none |
| `hyperhive.swarm_builder.queue.wait` | `s` | histogram | `kind` |
| `hyperhive.swarm_builder.publish.failures` | — | counter | none |
`kind` `pr` is an agent config PR's head and `meta` an agent built from the
swarm meta flake. Lock and archetype requests aren't counted. `refused` is a
request that waited out the one-build slot, which the builder then answers busy
without building, so it has a `builds` point and no `build.duration` point. A build that
can't start (no forge token) counts as a `failure` with a duration; a payload
that doesn't decode counts as a `failure` in `builds` only.
`build.duration` runs from holding the slot to the reply, including the cache
push of a meta build. `queue.wait` runs from the request's arrival to the slot,
or to the refusal. `queue.depth` is the requests waiting for the slot now,
exported every minute, 0 when idle, so it doubles as the liveness signal; the
`bao.client_cert.*` gauges below ride the same exporter.
`publish.failures` counts replies the queue refused to take, which the
controller sees as a missing answer.
## Store client-certificate metrics
Every service that logs in to the secret store with a client certificate can
report when that certificate stops being valid. Two observable gauges, one
data point each, unix seconds:
| Metric | Unit | Kind | Attributes |
| ---------------------------- | ---- | ----- | ---------- |
| `bao.client_cert.not_before` | `s` | gauge | `cert` |
| `bao.client_cert.not_after` | `s` | gauge | `cert` |
`cert` names the identity (`client`, `controller`, `agent`, `review-classifier`,
`ci-metrics`, `builder`); the resource labels (`service.name`, `hive`, `agent`) say who
holds it. `swarm_secret_client::cert_metrics` (cargo feature `otel`) registers both gauges on the service's
existing exporter and reads `BAO_CLIENT_CERT` at every collection, so a
renewed leaf shows up on the next export. A file that's missing, or that isn't a
certificate yields no data point and a debug log line. The parser is
`swarm_secret_client::cert`.
| Service | `cert` | Exporter it rides |
| ------------------------- | ------------------- | --------------------------------------------------- |
| `hive-c0re` | `client` | the container-stats exporter |
| `hive-agent` | `agent` | the per-turn exporter, built on its first recording |
| `swarm-controller` | `controller` | the job-graph exporter |
| `swarm-review-classifier` | `review-classifier` | the cost exporter, built on its first recording |
| `swarm-ci-metrics` | `ci-metrics` | a push of its own at the start of each timer pass |
| `swarm-builder` | `builder` | the build exporter, started with the daemon |
Not covered yet: the other six Rust services that hold a store certificate
(`swarm-claude-login`, `swarm-notify-router`,
`swarm-nats-auth`, `swarm-logs-auth`, `swarm-metrics-auth`,
`swarm-matrix-ctl`) have no exporter, and the ten identities that only shell
units use report nothing. Neither does any CA certificate. Each leaf's own
window is all the gauges show, and a CA that expires before its leaves is
invisible to them.
The swarm Grafana alerts on the lifetime fraction:
| Alert | Fires when |
| ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `hh-bao-client-cert-lifetime` | `(time() - not_before) / (not_after - not_before)` stays above 0.8 for 10 minutes, per `service.name`, `cert`, `hive` and `agent` |
The two gauges of a series carry identical labels, so MetricsQL divides them
one-to-one without `on()` or `group_left`. The query wraps each in
`last_over_time(...[15m])` so `swarm-ci-metrics`, which reports every five
minutes, keeps a series. A series that stops reporting resolves the alert
rather than firing it (no data counts as OK).
An agent leaf lives 7 days and renews at half its life, so a fraction over 0.8
means the renewal stalled about two days ago. The host leaves (`client`,
`controller`, and the others minted by `swarm-bao-pki`) live 3650 days and never
renew; for them 0.8 comes about eight years after minting. A guard on the days
left would suit them better and doesn't exist yet.
## Agent-emitted custom metrics (`hive-metric`)
Agents can push arbitrary labeled metrics to the same OTEL collector via the
`hive-metric` CLI tool, available in every agent container.
### Usage
```text
hive-metric <name> <value> [--type counter|gauge] [--temporality delta|cumulative] [--labels key=value...]
```
- `<name>` — metric name (for example `tasks_completed`, `latency_ms`).
- `<value>` — numeric value (f64; integers and floats both accepted).
- `--type counter|gauge` — metric kind: `counter` (increasing sum, default) or
`gauge` (instantaneous point-in-time value).
- `--temporality delta|cumulative` — counter reporting mode (`counter` only,
ignored for `gauge`): `delta` (this call's own contribution, default — send
`1` each time and the collector accumulates) or `cumulative` (this call
reports the running total, which a stateless one-shot CLI can't track
itself).
- `--labels key=value` — extra per-data-point labels. Repeat the flag for
more than one.
`hive-metric` inherits the resource labels (agent, swarm, service.name)
automatically from `OTEL_RESOURCE_ATTRIBUTES` — don't re-specify them.
### Examples
```text
# Counter: one more task finished (delta is the default — no flag needed)
hive-metric tasks_completed 1 --labels phase=scan
# Gauge: current queue depth (absolute value — must use --type gauge)
hive-metric queue_depth 17 --type gauge
# Float gauge with multiple labels (instantaneous measurement)
hive-metric api_latency_ms 142.5 --type gauge --labels model=sonnet --labels tier=api
```
### Error when an agent turns OTEL off
An agent whose own config sets `services.hyperhive.agent.otel.enable = false`
has no `OTEL_EXPORTER_OTLP_ENDPOINT` env var, and `hive-metric` exits with an
informative error message. No silently dropped metrics.
### Wire format
`hive-metric` always uses **OTLP HTTP/JSON** (`application/json` POST to
`$OTEL_EXPORTER_OTLP_ENDPOINT/v1/metrics`), regardless of the
`OTEL_EXPORTER_OTLP_PROTOCOL` setting. `hive-metric` forwards auth headers from
`OTEL_EXPORTER_OTLP_HEADERS` verbatim.
## Metrics temporality
The harness configures OTEL export with **cumulative** temporality by default
(`OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=cumulative`),
overriding Claude Code's default of DELTA. This avoids silent metric drops in
Prometheus-family backends (including Grafana LGTM / Mimir) that don't ship a
delta-to-cumulative processor.
<!-- vale write-good.Passive = NO -->
**`hive-metric` counters are the one exception**, reporting delta by default
(see above) — programmatically set on the exporter, which overrides this
container-wide env var for that tool specifically. `--type gauge` is
unaffected either way; gauges have no temporality. The hive-tier collector
runs a `deltatocumulative` processor ahead of export, so a delta
`hive-metric` counter still lands in VictoriaMetrics as a cumulative
series — the standard `rate()`/`increase()` idioms work on it exactly like
any other counter in this system, no extra query needed.
<!-- vale write-good.Passive = YES -->