hyperhive/docs/observability.md

# Observability (OpenTelemetry)

hyperhive has built-in support for exporting per-agent Claude Code statistics —
token usage, cost, tool call counts — to any OTLP-compatible collector via
Claude Code's built-in OpenTelemetry integration.

This is a **hive-wide** setting: one switch in the host NixOS config enables it
for every agent container simultaneously. There is no per-agent opt-in or opt-out.

## Enabling export

```nix
services.hyperhive.otel = {
  enable   = true;
  endpoint = "https://collector.example.com/otel";
};
```

`enable` is the single gate. `endpoint` (required when enabled) is the OTLP
HTTP endpoint; hive-c0re injects it as `OTEL_EXPORTER_OTLP_ENDPOINT` into
every agent's systemd service via the generated meta flake.

Each agent's harness (hive-ag3nt) exports directly to the collector — the
pipeline keeps working even when hive-c0re is down.

## Options reference

### `services.hyperhive.otel.enable` — bool, default `false`

Master switch. When true, all other options below take effect.

### `services.hyperhive.otel.endpoint` — string, required when enabled

OTLP collector endpoint URL. Set as `OTEL_EXPORTER_OTLP_ENDPOINT` for every
agent. Example: `"https://collector.example.com/otel"`.

### `services.hyperhive.otel.protocol` — enum, default `"http/protobuf"`

OTLP wire protocol, passed as `OTEL_EXPORTER_OTLP_PROTOCOL`. Accepted values:
- `"http/protobuf"` (default)
- `"http/json"`
- `"grpc"`

### `services.hyperhive.otel.headersCredential` — string or null, default `null`

Absolute path to a secret file on the host whose contents become
`OTEL_EXPORTER_OTLP_HEADERS` (e.g. `Authorization=Bearer <token>`).

hive-c0re forwards this host file into each agent container via
`systemd-nspawn --load-credential=otel-headers:<path>`; the inner harness unit
inherits it by name. The token is never copied into the nix store, generated
config, a bind mount, or argv.

Leave `null` if the endpoint needs no auth header. A configured-but-missing
file is skipped with a log warning — OTEL still exports, just without the auth
header.

```nix
services.hyperhive.otel = {
  enable           = true;
  endpoint         = "https://collector.example.com/otel";
  headersCredential = "/run/secrets/otel-headers";
};
```

### `services.hyperhive.otel.extraResourceAttributes` — string, default `""`

Extra comma-separated entries appended to `OTEL_RESOURCE_ATTRIBUTES` after the
built-in labels (`service.name`, `agent`, `hive`, `swarm`). Example:

```nix
extraResourceAttributes = "deployment.environment=prod,team=platform";
```

### `services.hyperhive.otel.debug` — bool, default `false`

When `true`, sets `CLAUDE_CODE_OTEL_DIAG_STDERR=1` in every agent container,
causing the OTEL SDK to emit diagnostic messages to stderr. Useful when
troubleshooting collector connectivity or endpoint config errors. Leave `false`
in normal operation — SDK errors from a misconfigured endpoint would otherwise
appear in every agent's journal unconditionally.

Only meaningful when `enable` is true.

### `services.hyperhive.otel.metricIntervalMs` — positive int or null, default `null`

Metric export interval in milliseconds, set as `OTEL_METRIC_EXPORT_INTERVAL`
for every agent. Claude Code's default is 60000 (60 s). Leave `null` to keep
that default.

Each agent runs claude as a short-lived per-turn process; claude force-flushes
metrics on process exit, so interval tuning is not required for metrics to be
exported. A lower value gives more frequent intermediate flushes within
long-running turns — cosmetic, not a correctness knob.

## Running a collector on the host (`otel.collector`)

By default every agent exports **straight to `endpoint`**, which means every
agent needs `headersCredential` to authenticate — and the harness delivers that
token into the agent's own `~/.claude/settings.json`, a file the agent can read.
`0600` protects it from other containers, not from the agent itself.

With `services.hyperhive.otel.collector.enable = true`, a collector runs on the
host and holds the credential instead. Agents export **unauthenticated** to a
bridge address only their own containers can reach; the collector adds the
upstream header and forwards to `endpoint`.

```nix
services.hyperhive.otel = {
  enable            = true;
  endpoint          = "https://collector.example.com/otel";  # still the upstream
  headersCredential = "/run/secrets/otel-headers";           # now only the host reads it
  collector.enable  = true;
};
```

**`endpoint` keeps meaning "where telemetry ultimately goes."** Turning the
collector on does not redefine it — the agent-facing value is *derived*
(`http://<bridgeIp>:<collector.port>`), so an existing deployment's `endpoint`
keeps working unchanged. The bridge port is contributed to
`exposeHostPorts` automatically; there is nothing to open by hand.

⚠️ **The trade:** without a collector each harness exports directly, so
telemetry survives anything host-side being down. A local collector is a new
dependency in that path. It runs on the same host as the agents, so the window
is small — but it is not zero.

### `services.hyperhive.otel.collector.enable` — bool, default `false`

Off means *absent*: no unit, no port, and `endpoint` keeps its current meaning
for every agent. Requires `headersCredential` to be set — a collector with no
credential is pure indirection, and an assertion says so rather than letting it
deploy.

### `services.hyperhive.otel.collector.port` — port, default `4318`

The OTLP/HTTP port the collector listens on, bound to the bridge IP only.

### `services.hyperhive.otel.collector.upstreamHeaderName` — string, default `"Authorization"`

Name of the header the collector sends upstream. The **value** comes from the
credential file at runtime (`EnvironmentFile` → `${env:<name>}`), never from
nix — so header names are config and header values are secrets, which is the
only split the collector's static header map can express.

⚠️ **`endpoint` must be valid for `protocol`.** The upstream exporter follows
`otel.protocol` (`grpc` → the gRPC exporter, otherwise OTLP/HTTP), and the gRPC
exporter takes an *address*: `https://host/path` is a legal
`OTEL_EXPORTER_OTLP_ENDPOINT` for HTTP but fails as gRPC with *"missing port in
address"*. The collector's config is validated at build time, so a mismatch is
a build error naming the reason rather than telemetry silently going nowhere.

## Network access

Agent containers can only reach the host on ports 80 and 443 by default. If
your OTLP collector runs on a non-standard port on the same host (e.g. a local
dev collector on `:4318`), open that port via:

```nix
services.hyperhive.network.exposeHostPorts = [ 4318 ];
```

Then point the endpoint at the bridge IP rather than loopback:

```nix
services.hyperhive.otel.endpoint = "http://10.42.0.1:4318";
```

The bridge IP is the host's address on the `hvbr0` bridge, typically
`10.42.0.1`. See `docs/network.md::Reaching host services` for details.

This is the manual form of what `otel.collector.enable` does for you — with the
collector on, the port is contributed and the endpoint derived, so neither line
above is needed.

## Built-in resource labels

Every agent's export includes these resource attributes automatically:

| Attribute | Value |
|-----------|-------|
| `service.name` | `hyperhive-agent` (constant) |
| `agent` | agent logical name (e.g. `iris`) |
| `hive` | hive display name (`services.hyperhive.hiveName`) |
| `swarm` | swarm display name (`services.hyperhive.swarm.name`, if set) |

Additional labels can be appended via `extraResourceAttributes` (see option
reference above); custom per-data-point labels can be passed with
`hive-metric --labels` (see below).

## Host-emitted container-resource metrics (hive-c0re)

When OTEL is enabled, **hive-c0re itself** also exports each agent
container's resource load — the same cgroup gauges shown on the dashboard
LOAD tab — to the configured `endpoint`, reusing the same
`services.hyperhive.otel` config (no separate toggle). These come from the
host, not the in-container Claude SDK, so they cover containers even when
their agent is idle.

Emitted via the OpenTelemetry Rust SDK, using the
[semconv `container.*`](https://opentelemetry.io/docs/specs/semconv/system/container-metrics/)
metric names + the standard `container.name` attribute where a spec metric
exists, so off-the-shelf OTEL/Grafana container dashboards work. Resource
`service.name = hyperhive-c0re`; each data point is tagged `container.name`
(= the `h-<agent>` machine) and the hive `agent` label:

| Metric | Unit | Kind | Source |
|--------|------|------|--------|
| `container.cpu.time` | `s` | counter | cumulative `cpu.stat` `usage_usec` → seconds |
| `container.memory.usage` | `By` | gauge | `memory.current` |
| `hyperhive.container.memory.limit` | `By` | gauge | `memory.max` (custom — semconv has no `.limit` metric; omitted when unlimited) |
| `hyperhive.container.memory.peak` | `By` | gauge | `memory.peak` (custom — no semconv metric; omitted if unavailable) |
| `hyperhive.container.storage.usage` | `By` | gauge | state dir + writable rootfs (custom — semconv only has `disk.io`; omitted until the slow disk sampler runs) |
| `hyperhive.container.cpu.percent` | `%` | gauge | host-normalised percent (custom — the value the dashboard LOAD tab shows, no `rate()` needed) |

The `hyperhive.`-prefixed metrics have no semconv equivalent (memory
limit + peak, on-disk footprint, and an instantaneous cpu percent kept
alongside the spec `container.cpu.time` counter for convenience). Hive
labels (`hive`, `swarm`, …) ride on the resource via
`extraResourceAttributes`.

Cadence follows `metricIntervalMs` (default 60s). Transport is OTLP/HTTP
(JSON) to `<endpoint>`; the auth header is loaded onto hive-c0re's own unit
via systemd `LoadCredential` (from the same `headersCredential` file) and
sent as `Authorization`.

## Agent-emitted custom metrics (`hive-metric`)

Agents can push arbitrary labeled metrics to the same OTEL collector via the
`hive-metric` CLI tool, available in every agent container when
`services.hyperhive.otel.enable = true`.

### Usage

```text
hive-metric <name> <value> [--type counter|gauge] [--labels key=value...]
```

- `<name>` — metric name (e.g. `tasks_completed`, `latency_ms`).
- `<value>` — numeric value (f64; integers and floats both accepted).
- `--type counter|gauge` — metric kind: `counter` (cumulative sum, default) or
  `gauge` (instantaneous point-in-time value).
- `--labels key=value` — extra per-data-point labels. May be repeated.
  The resource labels (agent, hive, swarm, service.name) are inherited
  automatically from `OTEL_RESOURCE_ATTRIBUTES` — do not re-specify them.

### Examples

```text
# Counter: cumulative tasks finished (default type — no --type flag needed)
hive-metric tasks_completed 1 --labels phase=scan

# Gauge: current queue depth (absolute value — must use --type gauge)
hive-metric queue_depth 17 --type gauge

# Float gauge with multiple labels (instantaneous measurement)
hive-metric api_latency_ms 142.5 --type gauge --labels model=sonnet --labels tier=api
```

### Error when OTEL is not configured

When `services.hyperhive.otel.enable = false` (the default), the
`OTEL_EXPORTER_OTLP_ENDPOINT` env var is not set and `hive-metric` exits
with an informative error message. No silently-dropped metrics.

### Wire format

`hive-metric` always uses **OTLP HTTP/JSON** (`application/json` POST to
`$OTEL_EXPORTER_OTLP_ENDPOINT/v1/metrics`), regardless of the
`OTEL_EXPORTER_OTLP_PROTOCOL` setting. Auth headers from
`OTEL_EXPORTER_OTLP_HEADERS` are forwarded verbatim.

## Metrics temporality

OTEL export is always configured with **cumulative** temporality
(`OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=cumulative`),
overriding Claude Code's default of DELTA. This avoids silent metric drops in
Prometheus-family backends (including Grafana LGTM / Mimir) that don't ship a
delta-to-cumulative processor.