From 174876094ead70687c0c337a9b52823b63074d62 Mon Sep 17 00:00:00 2001 From: iris Date: Sat, 4 Jul 2026 13:14:53 +0200 Subject: [PATCH] docs(observability): document OTEL configuration options MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add docs/observability.md covering all services.hyperhive.otel.* options: enable, endpoint, protocol, headersCredential, extraResourceAttributes, debug (new in cb0a66147a5f), and metricIntervalMs. Includes: - Built-in OTEL_RESOURCE_ATTRIBUTES labels (service.name, agent, hive, swarm) - Cumulative temporality note (avoids Prometheus DELTA drop) - Network note for host-side collectors on non-standard ports, cross-referencing docs/network.md exposeHostPorts Also: - CLAUDE.md: add reading-path entry for the new doc - docs/network.md: link the OTEL mention to observability.md Closes no issue — gap found during doc sweep. --- CLAUDE.md | 3 + docs/network.md | 2 +- docs/observability.md | 134 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 138 insertions(+), 1 deletion(-) create mode 100644 docs/observability.md diff --git a/CLAUDE.md b/CLAUDE.md index b200aacb..86f4913f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -113,6 +113,9 @@ read them à la carte. - **"What is `/knowledge`? How does the hive-wide knowledge repo sync, and how do I contribute a document?"** → [`docs/knowledge.md`](docs/knowledge.md). +- **"How do I export Claude Code metrics (tokens, cost, tool calls) to + a Prometheus/Grafana collector? What OTEL options are available?"** → + [`docs/observability.md`](docs/observability.md). ## Conventions & process diff --git a/docs/network.md b/docs/network.md index 81ba505f..3fad6e3e 100644 --- a/docs/network.md +++ b/docs/network.md @@ -105,7 +105,7 @@ and any other HTTP services. By default agents can only reach the host on 80/443 (+53 DNS), so a host-side service on another port — e.g. a dev OTEL collector for -`services.hyperhive.otel.endpoint` — is unreachable. +`services.hyperhive.otel.endpoint` (see `docs/observability.md`) — is unreachable. `services.hyperhive.network.exposeHostPorts = [ 4318 ];` opens each listed TCP port `P` on the bridge-interface `allowedTCPPorts`, so an diff --git a/docs/observability.md b/docs/observability.md new file mode 100644 index 00000000..25e7e358 --- /dev/null +++ b/docs/observability.md @@ -0,0 +1,134 @@ +# Observability (OpenTelemetry) + +hyperhive has built-in support for exporting per-agent Claude Code statistics — +token usage, cost, tool call counts — to any OTLP-compatible collector via +Claude Code's built-in OpenTelemetry integration. + +This is a **hive-wide** setting: one switch in the host NixOS config enables it +for every agent container simultaneously. There is no per-agent opt-in or opt-out. + +## Enabling export + +```nix +services.hyperhive.otel = { + enable = true; + endpoint = "https://collector.example.com/otel"; +}; +``` + +`enable` is the single gate. `endpoint` (required when enabled) is the OTLP +HTTP endpoint; hive-c0re injects it as `OTEL_EXPORTER_OTLP_ENDPOINT` into +every agent's systemd service via the generated meta flake. + +Each agent's harness (hive-ag3nt) exports directly to the collector — the +pipeline keeps working even when hive-c0re is down. + +## Options reference + +### `services.hyperhive.otel.enable` — bool, default `false` + +Master switch. When true, all other options below take effect. + +### `services.hyperhive.otel.endpoint` — string, required when enabled + +OTLP collector endpoint URL. Set as `OTEL_EXPORTER_OTLP_ENDPOINT` for every +agent. Example: `"https://collector.example.com/otel"`. + +### `services.hyperhive.otel.protocol` — enum, default `"http/protobuf"` + +OTLP wire protocol, passed as `OTEL_EXPORTER_OTLP_PROTOCOL`. Accepted values: +- `"http/protobuf"` (default) +- `"http/json"` +- `"grpc"` + +### `services.hyperhive.otel.headersCredential` — string or null, default `null` + +Absolute path to a secret file on the host whose contents become +`OTEL_EXPORTER_OTLP_HEADERS` (e.g. `Authorization=Bearer `). + +hive-c0re forwards this host file into each agent container via +`systemd-nspawn --load-credential=otel-headers:`; the inner harness unit +inherits it by name. The token is never copied into the nix store, generated +config, a bind mount, or argv. + +Leave `null` if the endpoint needs no auth header. A configured-but-missing +file is skipped with a log warning — OTEL still exports, just without the auth +header. + +```nix +services.hyperhive.otel = { + enable = true; + endpoint = "https://collector.example.com/otel"; + headersCredential = "/run/secrets/otel-headers"; +}; +``` + +### `services.hyperhive.otel.extraResourceAttributes` — string, default `""` + +Extra comma-separated entries appended to `OTEL_RESOURCE_ATTRIBUTES` after the +built-in labels (`service.name`, `agent`, `hive`, `swarm`). Example: + +```nix +extraResourceAttributes = "deployment.environment=prod,team=platform"; +``` + +### `services.hyperhive.otel.debug` — bool, default `false` + +When `true`, sets `CLAUDE_CODE_OTEL_DIAG_STDERR=1` in every agent container, +causing the OTEL SDK to emit diagnostic messages to stderr. Useful when +troubleshooting collector connectivity or endpoint config errors. Leave `false` +in normal operation — SDK errors from a misconfigured endpoint would otherwise +appear in every agent's journal unconditionally. + +Only meaningful when `enable` is true. + +### `services.hyperhive.otel.metricIntervalMs` — positive int or null, default `null` + +Metric export interval in milliseconds, set as `OTEL_METRIC_EXPORT_INTERVAL` +for every agent. Claude Code's default is 60000 (60 s). Leave `null` to keep +that default. + +Each agent runs claude as a short-lived per-turn process; claude force-flushes +metrics on process exit, so interval tuning is not required for metrics to be +exported. A lower value gives more frequent intermediate flushes within +long-running turns — cosmetic, not a correctness knob. + +## Network access + +Agent containers can only reach the host on ports 80 and 443 by default. If +your OTLP collector runs on a non-standard port on the same host (e.g. a local +dev collector on `:4318`), open that port via: + +```nix +services.hyperhive.network.exposeHostPorts = [ 4318 ]; +``` + +Then point the endpoint at the bridge IP rather than loopback: + +```nix +services.hyperhive.otel.endpoint = "http://10.42.0.1:4318"; +``` + +The bridge IP is the host's address on the `hvbr0` bridge, typically +`10.42.0.1`. See `docs/network.md::Reaching host services` for details. + +## Built-in resource labels + +Every agent's export includes these resource attributes automatically: + +| Attribute | Value | +|-----------|-------| +| `service.name` | `hyperhive-agent` (constant) | +| `agent` | agent logical name (e.g. `iris`) | +| `hive` | hive display name (`services.hyperhive.hiveName`) | +| `swarm` | swarm display name (`services.hyperhive.swarmName`, if set) | + +Additional labels can be appended via `extraResourceAttributes`. + +## Metrics temporality + +OTEL export is always configured with **cumulative** temporality +(`OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=cumulative`), +overriding Claude Code's default of DELTA. This avoids silent metric drops in +Prometheus-family backends (including Grafana LGTM / Mimir) that don't ship a +delta-to-cumulative processor.