hyperhive/docs/observability.md

Observability (OpenTelemetry)

hyperhive has built-in support for exporting per-agent Claude Code statistics — token usage, cost, tool call counts — to any OTLP-compatible collector via Claude Code's built-in OpenTelemetry integration.

This is a hive-wide setting: one switch in the host NixOS config enables it for every agent container simultaneously. There is no per-agent opt-in or opt-out.

Enabling export

services.hyperhive.otel = {
  enable   = true;
  endpoint = "https://collector.example.com/otel";
};

enable is the single gate. endpoint is where telemetry ends up after it leaves the swarm — optional, because the swarm's own metrics store (swarm.victoriametrics.enable) is a destination in its own right. With both, telemetry goes to both. See swarm/services.md.

There is exactly one way telemetry leaves a hive: through the collector that enable starts on the host. Agents never talk to endpoint themselves — they export unauthenticated to a bridge address only their own containers can reach. That collector forwards to the swarm's (swarm/services.md), which is the single process holding the upstream credential and the only writer to the swarm's store. No agent holds a copy, and neither does this hive.

The hive collector reaches the swarm collector by its gateway name (swarm.otel.domain, default otel.<swarm domain>) — the same DNS-and-CA-trust shape every hive-to-swarm-service hop uses, not a URL an operator has to point anywhere. A hive that does not run the swarm's services still resolves that name through the gateway; nothing here needs setting for the split-host case.

⚠️ The collector is therefore in the path of all telemetry. It runs on the same host as the agents and restarts on failure, and telemetry is not the control plane — degraded telemetry is not degraded operation — but the export no longer survives independently of anything host-side.

what the agent→collector hop is and isn't

It has no application-level auth. The receiver takes any OTLP that reaches it; what bounds who can reach it is the firewall — exposeHostPorts opens the port on the bridge interface only. So "unauthenticated to a bridge address" means reachable from an agent container, not presents a credential.

The consequence, stated because it is a choice rather than an oversight: any agent can push arbitrary OTLP, and it is forwarded on under the operator's credential. Neither tier can tell a container's genuine Claude Code stats from anything else shaped like OTLP arriving on that port — including data smuggled out in resource attributes on an otherwise-legitimate export.

That is a different risk from the one the collector fixes, and strictly smaller than what preceded it: before, every agent held the upstream credential itself, so it could do all of the above and use the token anywhere else. The collector removes the token and keeps the pipe. Agents are inside the trust boundary (docs/security.md: capability = accepted risk), so an agent being able to send is an accepted extension of that boundary — but it is not closed by this design, and nothing here should be read as closing it.

Options reference

Every services.hyperhive.otel.* option's full type/default/description/ example is generated straight from the nix module (nix/host-modules/ otel.nix) into /options/ (host options — nix build .#docs-host for a local render). That page is kept honest by the build in a way a hand-copied version here cannot be, so it is the reference, not this doc. What follows is what a flat per-option listing can't express: the two-tier architecture, the security model, and how the options interact.

The two collectors

Telemetry crosses two collectors, and which one you configure depends on what the host is:

	runs where	receives from	does
hive tier — otel.enable	every hive with agents	that hive's agents, on the bridge	forwards to the swarm tier. Holds no credential, picks no destination
swarm tier — swarm.otel.enable	once per swarm	every hive's collector	writes the swarm's store and exports upstream

An all-local host runs both, and needs nothing said about the hop between them.

services.hyperhive.otel = {
  enable            = true;
  endpoint          = "https://collector.example.com/otel";  # the upstream
  headersCredential = "/run/secrets/otel-headers";           # only the swarm tier reads it
};

Why the hive tier isn't optional. Exporting straight to endpoint means every agent needs the credential to authenticate — and the harness delivers that token into the agent's own ~/.claude/settings.json, a file the agent can read. 0600 protects it from other containers, not from the agent itself. As long as the direct path stays selectable, that hole stays selectable; an option that can reintroduce it is a hole with extra steps.

Why the tiers stay separate on one box. They are not collapsed when co-located: an all-local hive is a statement about where processes run, not about the shape of the deployment. A boundary that disappears locally is one the local deployment stops testing.

endpoint keeps meaning "where telemetry goes upstream." Neither tier redefines it — the agent-facing value is derived (http://<bridgeIp>:<collector.port>), so an existing deployment's endpoint keeps working unchanged. The bridge port is contributed to exposeHostPorts automatically; there is nothing to open by hand.

Authenticated ingest

The swarm tier gives each hive its own receiver, and stamps the hive label from whichever receiver accepted a sample. A hive therefore cannot report metrics as another hive, and cannot relabel its own by editing what it sends — the label is not taken from the payload at all.

On an all-local swarm there is nothing to set. Each hive already has an identity, and its collector reads the secret that host's own authelia minted.

On a hive that does not host the swarm's services, the secret has to arrive somehow — copy it across and name it:

services.hyperhive.otel.clientSecretFile = "/run/secrets/hive-telemetry.secret";

There is no unauthenticated mode. A hive always presents an identity, so a missing credential is a build error rather than a quieter fallback — the collector has no anonymous route to accept samples on, and every path it serves belongs to exactly one hive.

Getting the secret wrong shows up as the hive's collector logging 401s from the swarm tier and no metrics appearing for that hive.

⚠️ endpoint must be valid for protocol. The upstream exporter follows otel.protocol (grpc → the gRPC exporter, otherwise OTLP/HTTP), and the gRPC exporter takes an address: https://host/path is a legal OTEL_EXPORTER_OTLP_ENDPOINT for HTTP but fails as gRPC with "missing port in address". The collector's config is validated at build time, so a mismatch is a build error naming the reason rather than telemetry silently going nowhere.

Network access

Agent containers can only reach the host on ports 80 and 443 by default. To let them reach some other host-local service you run yourself — a database, a scratch HTTP endpoint — open its port on the bridge:

services.hyperhive.network.exposeHostPorts = [ 5432 ];

and point whatever consumes it at 10.42.0.1:5432 rather than loopback: inside a container, loopback is the container. The bridge IP is the host's address on the hive-br0 bridge. The service must also bind an address the bridge can reach — a 127.0.0.1-only listener stays unreachable no matter what the firewall allows. See docs/network.md::Reaching host services for details.

⚠️ None of this is needed for hyperhive's own telemetry — otel.enable contributes the collector's port and derives the agent-facing endpoint itself.

Built-in resource labels

The OTLP variables (OTEL_EXPORTER_OTLP_ENDPOINT, _PROTOCOL, OTEL_RESOURCE_ATTRIBUTES, the temporality preference) are set container wide — in systemd's DefaultEnvironment and in /etc/profile — so every process in an agent container exports to the hive's collector without any per-tool wiring. That covers Claude Code, hive-metric, and anything you run yourself from a tool call or hivectl shell.

Every agent's export therefore includes these resource attributes automatically:

Attribute	Value
service.name	hyperhive-agent (constant)
agent	agent logical name (e.g. iris)
hive	hive display name (services.hyperhive.hiveName)
swarm	swarm display name (services.hyperhive.swarm.name, if set)

Additional labels can be appended via extraResourceAttributes (see option reference above); custom per-data-point labels can be passed with hive-metric --labels (see below).

Host-emitted container-resource metrics (hive-c0re)

When OTEL is enabled, hive-c0re itself also exports each agent container's resource load — the same cgroup gauges shown on the dashboard LOAD tab — to this hive's own collector, exactly like an agent does and with no separate toggle. These come from the host, not the in-container Claude SDK, so they cover containers even when their agent is idle.

Emitted via the OpenTelemetry Rust SDK, using the semconv container.* metric names + the standard container.name attribute where a spec metric exists, so off-the-shelf OTEL/Grafana container dashboards work. Resource service.name = hyperhive-c0re; each data point is tagged container.name (= the h-<agent> machine) and the hive agent label:

Metric	Unit	Kind	Source
container.cpu.time	s	counter	cumulative cpu.stat usage_usec → seconds
container.memory.usage	By	gauge	memory.current
hyperhive.container.memory.limit	By	gauge	memory.max (custom — semconv has no .limit metric; omitted when unlimited)
hyperhive.container.memory.peak	By	gauge	memory.peak (custom — no semconv metric; omitted if unavailable)
hyperhive.container.storage.usage	By	gauge	state dir + writable rootfs (custom — semconv only has disk.io; omitted until the slow disk sampler runs)
hyperhive.container.cpu.percent	%	gauge	host-normalised percent (custom — the value the dashboard LOAD tab shows, no rate() needed)

The hyperhive.-prefixed metrics have no semconv equivalent (memory limit + peak, on-disk footprint, and an instantaneous cpu percent kept alongside the spec container.cpu.time counter for convenience). Hive labels (hive, swarm, …) ride on the resource via extraResourceAttributes.

Cadence follows metricIntervalMs (default 60s). Transport is OTLP/HTTP (JSON) to the hive collector's bridge address, with no auth header — that first hop is unauthenticated for every producer on this host, and the upstream credential stays on the swarm tier.

Agent-emitted per-turn metrics (hive-agent)

When OTEL is enabled, the harness itself (hive-agent) exports one small set of metrics per claude turn, recorded the moment the turn ends (not polled). These are deliberately the fields Claude Code's own built-in export (see above) can't know about — the harness's own wall-clock timing, what woke the turn, its own outcome classification, the loose-ends backlog, and session boundaries. Token usage, cost, and tool-call counts are not duplicated here; that's already covered by Claude's own export.

Metric	Unit	Kind	Attributes
hyperhive.agent.turn.duration	ms	histogram	wake_from, result_kind, model
hyperhive.agent.turn.count	—	counter	wake_from, result_kind, model
hyperhive.agent.session.count	—	counter	model (incremented once per fresh, non---continue'd session)
hyperhive.agent.loose_ends.threads	—	gauge	none
hyperhive.agent.loose_ends.reminders	—	gauge	none

Resource attributes (service.name, agent, hive, swarm) come from the same container-wide OTEL_RESOURCE_ATTRIBUTES as everything else in this section — nothing extra to configure. Cadence follows HYPERHIVE_OTEL_METRIC_INTERVAL_MS (default 60s, same variable + default as hive-c0re's container-resource export above) — that only controls how often the batched points are flushed to the collector, not how often they're recorded (every turn, always).

Hive-scoped metrics (hive-c0re)

Everything above is measured per agent, tagged with the hive it runs in. These three are measured per hive, and carry no agent label — so a hive that hosts no agents still reports, and "this hive is quiet" is distinguishable from "this hive is gone". Select them with {hive!="",agent=""}.

Metric	Unit	Kind	Meaning
process.uptime	s	gauge	seconds since this hive's hive-c0re started exporting; a restart reads as a drop to ~0
hyperhive.hive.degraded	1	gauge	1 while the hive reports itself unhealthy — the same verdict /health/ready gives and the swarm status view shows
hyperhive.hive.warnings	1	gauge	how many warnings are currently raised, split by a level attribute (warn, crit)

Both levels are reported every cycle, 0 included, so a healthy hive is visible as zeros rather than as missing series.

hyperhive.hive.degraded is what a dashboard should alert on: it is hive-c0re's own readiness verdict, so it stays in step with /health/ready and with what the swarm controller sees. hyperhive.hive.warnings is the detail behind it — warn-level entries mean "an operator should look" and do not set degraded.

Same cadence, transport and resource labels as the container metrics above.

Agent-emitted custom metrics (hive-metric)

Agents can push arbitrary labeled metrics to the same OTEL collector via the hive-metric CLI tool, available in every agent container when services.hyperhive.otel.enable = true.

Usage

hive-metric <name> <value> [--type counter|gauge] [--labels key=value...]

• <name> — metric name (e.g. tasks_completed, latency_ms).
• <value> — numeric value (f64; integers and floats both accepted).
• --type counter|gauge — metric kind: counter (cumulative sum, default) or gauge (instantaneous point-in-time value).
• --labels key=value — extra per-data-point labels. May be repeated. The resource labels (agent, hive, swarm, service.name) are inherited automatically from OTEL_RESOURCE_ATTRIBUTES — do not re-specify them.

Examples

# Counter: cumulative tasks finished (default type — no --type flag needed)
hive-metric tasks_completed 1 --labels phase=scan

# Gauge: current queue depth (absolute value — must use --type gauge)
hive-metric queue_depth 17 --type gauge

# Float gauge with multiple labels (instantaneous measurement)
hive-metric api_latency_ms 142.5 --type gauge --labels model=sonnet --labels tier=api

Error when OTEL is not configured

When services.hyperhive.otel.enable = false (the default), the OTEL_EXPORTER_OTLP_ENDPOINT env var is not set and hive-metric exits with an informative error message. No silently-dropped metrics.

Wire format

hive-metric always uses OTLP HTTP/JSON (application/json POST to $OTEL_EXPORTER_OTLP_ENDPOINT/v1/metrics), regardless of the OTEL_EXPORTER_OTLP_PROTOCOL setting. Auth headers from OTEL_EXPORTER_OTLP_HEADERS are forwarded verbatim.

Metrics temporality

OTEL export is always configured with cumulative temporality (OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=cumulative), overriding Claude Code's default of DELTA. This avoids silent metric drops in Prometheus-family backends (including Grafana LGTM / Mimir) that don't ship a delta-to-cumulative processor.