docs(observability): document OTEL configuration options
Add docs/observability.md covering all services.hyperhive.otel.*
options: enable, endpoint, protocol, headersCredential,
extraResourceAttributes, debug (new in cb0a66147a), and
metricIntervalMs.
Includes:
- Built-in OTEL_RESOURCE_ATTRIBUTES labels (service.name, agent, hive, swarm)
- Cumulative temporality note (avoids Prometheus DELTA drop)
- Network note for host-side collectors on non-standard ports,
cross-referencing docs/network.md exposeHostPorts
Also:
- CLAUDE.md: add reading-path entry for the new doc
- docs/network.md: link the OTEL mention to observability.md
Closes no issue — gap found during doc sweep.
This commit is contained in:
parent
2f8e9e2bb4
commit
174876094e
3 changed files with 138 additions and 1 deletions
|
|
@ -113,6 +113,9 @@ read them à la carte.
|
|||
- **"What is `/knowledge`? How does the hive-wide knowledge repo sync,
|
||||
and how do I contribute a document?"** →
|
||||
[`docs/knowledge.md`](docs/knowledge.md).
|
||||
- **"How do I export Claude Code metrics (tokens, cost, tool calls) to
|
||||
a Prometheus/Grafana collector? What OTEL options are available?"** →
|
||||
[`docs/observability.md`](docs/observability.md).
|
||||
|
||||
## Conventions & process
|
||||
|
||||
|
|
|
|||
|
|
@ -105,7 +105,7 @@ and any other HTTP services.
|
|||
|
||||
By default agents can only reach the host on 80/443 (+53 DNS), so a
|
||||
host-side service on another port — e.g. a dev OTEL collector for
|
||||
`services.hyperhive.otel.endpoint` — is unreachable.
|
||||
`services.hyperhive.otel.endpoint` (see `docs/observability.md`) — is unreachable.
|
||||
|
||||
`services.hyperhive.network.exposeHostPorts = [ 4318 ];` opens each
|
||||
listed TCP port `P` on the bridge-interface `allowedTCPPorts`, so an
|
||||
|
|
|
|||
134
docs/observability.md
Normal file
134
docs/observability.md
Normal file
|
|
@ -0,0 +1,134 @@
|
|||
# Observability (OpenTelemetry)
|
||||
|
||||
hyperhive has built-in support for exporting per-agent Claude Code statistics —
|
||||
token usage, cost, tool call counts — to any OTLP-compatible collector via
|
||||
Claude Code's built-in OpenTelemetry integration.
|
||||
|
||||
This is a **hive-wide** setting: one switch in the host NixOS config enables it
|
||||
for every agent container simultaneously. There is no per-agent opt-in or opt-out.
|
||||
|
||||
## Enabling export
|
||||
|
||||
```nix
|
||||
services.hyperhive.otel = {
|
||||
enable = true;
|
||||
endpoint = "https://collector.example.com/otel";
|
||||
};
|
||||
```
|
||||
|
||||
`enable` is the single gate. `endpoint` (required when enabled) is the OTLP
|
||||
HTTP endpoint; hive-c0re injects it as `OTEL_EXPORTER_OTLP_ENDPOINT` into
|
||||
every agent's systemd service via the generated meta flake.
|
||||
|
||||
Each agent's harness (hive-ag3nt) exports directly to the collector — the
|
||||
pipeline keeps working even when hive-c0re is down.
|
||||
|
||||
## Options reference
|
||||
|
||||
### `services.hyperhive.otel.enable` — bool, default `false`
|
||||
|
||||
Master switch. When true, all other options below take effect.
|
||||
|
||||
### `services.hyperhive.otel.endpoint` — string, required when enabled
|
||||
|
||||
OTLP collector endpoint URL. Set as `OTEL_EXPORTER_OTLP_ENDPOINT` for every
|
||||
agent. Example: `"https://collector.example.com/otel"`.
|
||||
|
||||
### `services.hyperhive.otel.protocol` — enum, default `"http/protobuf"`
|
||||
|
||||
OTLP wire protocol, passed as `OTEL_EXPORTER_OTLP_PROTOCOL`. Accepted values:
|
||||
- `"http/protobuf"` (default)
|
||||
- `"http/json"`
|
||||
- `"grpc"`
|
||||
|
||||
### `services.hyperhive.otel.headersCredential` — string or null, default `null`
|
||||
|
||||
Absolute path to a secret file on the host whose contents become
|
||||
`OTEL_EXPORTER_OTLP_HEADERS` (e.g. `Authorization=Bearer <token>`).
|
||||
|
||||
hive-c0re forwards this host file into each agent container via
|
||||
`systemd-nspawn --load-credential=otel-headers:<path>`; the inner harness unit
|
||||
inherits it by name. The token is never copied into the nix store, generated
|
||||
config, a bind mount, or argv.
|
||||
|
||||
Leave `null` if the endpoint needs no auth header. A configured-but-missing
|
||||
file is skipped with a log warning — OTEL still exports, just without the auth
|
||||
header.
|
||||
|
||||
```nix
|
||||
services.hyperhive.otel = {
|
||||
enable = true;
|
||||
endpoint = "https://collector.example.com/otel";
|
||||
headersCredential = "/run/secrets/otel-headers";
|
||||
};
|
||||
```
|
||||
|
||||
### `services.hyperhive.otel.extraResourceAttributes` — string, default `""`
|
||||
|
||||
Extra comma-separated entries appended to `OTEL_RESOURCE_ATTRIBUTES` after the
|
||||
built-in labels (`service.name`, `agent`, `hive`, `swarm`). Example:
|
||||
|
||||
```nix
|
||||
extraResourceAttributes = "deployment.environment=prod,team=platform";
|
||||
```
|
||||
|
||||
### `services.hyperhive.otel.debug` — bool, default `false`
|
||||
|
||||
When `true`, sets `CLAUDE_CODE_OTEL_DIAG_STDERR=1` in every agent container,
|
||||
causing the OTEL SDK to emit diagnostic messages to stderr. Useful when
|
||||
troubleshooting collector connectivity or endpoint config errors. Leave `false`
|
||||
in normal operation — SDK errors from a misconfigured endpoint would otherwise
|
||||
appear in every agent's journal unconditionally.
|
||||
|
||||
Only meaningful when `enable` is true.
|
||||
|
||||
### `services.hyperhive.otel.metricIntervalMs` — positive int or null, default `null`
|
||||
|
||||
Metric export interval in milliseconds, set as `OTEL_METRIC_EXPORT_INTERVAL`
|
||||
for every agent. Claude Code's default is 60000 (60 s). Leave `null` to keep
|
||||
that default.
|
||||
|
||||
Each agent runs claude as a short-lived per-turn process; claude force-flushes
|
||||
metrics on process exit, so interval tuning is not required for metrics to be
|
||||
exported. A lower value gives more frequent intermediate flushes within
|
||||
long-running turns — cosmetic, not a correctness knob.
|
||||
|
||||
## Network access
|
||||
|
||||
Agent containers can only reach the host on ports 80 and 443 by default. If
|
||||
your OTLP collector runs on a non-standard port on the same host (e.g. a local
|
||||
dev collector on `:4318`), open that port via:
|
||||
|
||||
```nix
|
||||
services.hyperhive.network.exposeHostPorts = [ 4318 ];
|
||||
```
|
||||
|
||||
Then point the endpoint at the bridge IP rather than loopback:
|
||||
|
||||
```nix
|
||||
services.hyperhive.otel.endpoint = "http://10.42.0.1:4318";
|
||||
```
|
||||
|
||||
The bridge IP is the host's address on the `hvbr0` bridge, typically
|
||||
`10.42.0.1`. See `docs/network.md::Reaching host services` for details.
|
||||
|
||||
## Built-in resource labels
|
||||
|
||||
Every agent's export includes these resource attributes automatically:
|
||||
|
||||
| Attribute | Value |
|
||||
|-----------|-------|
|
||||
| `service.name` | `hyperhive-agent` (constant) |
|
||||
| `agent` | agent logical name (e.g. `iris`) |
|
||||
| `hive` | hive display name (`services.hyperhive.hiveName`) |
|
||||
| `swarm` | swarm display name (`services.hyperhive.swarmName`, if set) |
|
||||
|
||||
Additional labels can be appended via `extraResourceAttributes`.
|
||||
|
||||
## Metrics temporality
|
||||
|
||||
OTEL export is always configured with **cumulative** temporality
|
||||
(`OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=cumulative`),
|
||||
overriding Claude Code's default of DELTA. This avoids silent metric drops in
|
||||
Prometheus-family backends (including Grafana LGTM / Mimir) that don't ship a
|
||||
delta-to-cumulative processor.
|
||||
Loading…
Reference in a new issue