# Observability (OpenTelemetry) hyperhive has built-in support for exporting each agent container's telemetry to any OTLP-compatible collector: Claude Code statistics — token usage, cost, tool call counts — via Claude Code's built-in OpenTelemetry integration, and the container's journal, forwarded by a collector running inside it. This is a **hive-wide** setting: one switch in the host NixOS config enables it for every agent container simultaneously. No per-agent opt-in or opt-out exists. ## Enabling export ```nix services.hyperhive.otel = { enable = true; endpoint = "https://collector.example.com/otel"; }; ``` `enable` is the single gate. `endpoint` is where telemetry ends up after it leaves the swarm — optional, because the swarm's own metrics store (`deploy.victoriametrics`) is a destination in its own right. With both, telemetry goes to both. See [`swarm/services.md`](../swarm/services.md#metrics-victoriametrics--grafana). **Telemetry leaves a hive exactly one way: through the collector that `enable` starts on the host.** Agents never talk to `endpoint` themselves — they export unauthenticated to a bridge address only their own containers can reach. That collector forwards to the swarm's ([`swarm/services.md`](../swarm/services.md#telemetry-collector-otel)), which is the single process holding the upstream credential and the only writer to the swarm's store. No agent holds a copy, and neither does this hive. The hive collector reaches the swarm collector by its gateway name (`swarm.otel.domain`, default `otel.`) — the same DNS-and-CA-trust shape every hive-to-swarm-service hop uses, not a URL an operator has to point anywhere. A hive that doesn't run the swarm's services still resolves that name through the gateway; nothing here needs setting for the split-host case. ⚠️ **The collector is therefore in the path of all telemetry.** It runs on the same host as the agents and restarts on failure, and telemetry isn't the control plane — degraded telemetry isn't degraded operation — but the export no longer survives independently of anything host-side. ### what the agent→collector hop is and isn't **It has no application-level auth.** The receiver takes any OTLP that reaches it; what bounds who can reach it's the firewall — `exposeHostPorts` opens the port on the bridge interface only — "unauthenticated to a bridge address" means _reachable from an agent container_, not _presents a credential_. The consequence, stated because it's a choice rather than an oversight: **any agent can push arbitrary OTLP, and it's forwarded on under the operator's credential.** Neither tier can tell a container's genuine Claude Code stats from anything else shaped like OTLP arriving on that port — including data smuggled out in resource attributes on an otherwise-legitimate export. That's a **different risk from the one the collector fixes**, and strictly smaller than what preceded it: before, every agent held the upstream credential itself, so it could do all of the above _and_ use the token anywhere else. The collector removes the token and keeps the pipe. Agents are inside the trust boundary (`docs/trust-boundary/security.md`: capability = accepted risk), so an agent being able to _send_ is an accepted extension of that boundary — but it's not closed by this design, and don't read anything here as closing it. **The `agent` label is self-reported, and the hive→swarm authentication below doesn't change that.** Treat it as a convenience for grouping dashboards, never as evidence of which container produced a sample: any agent that can reach this hive's collector can label its data as any other agent. **Logs ride the same hop, and a journal carries more than a counter does.** Each agent container forwards its own journal through this port — every unit in it at `info` and above, not an allowlist. That's the harness, the MCP daemons and whatever a tool call spawned, so command lines and error text now leave the container where before only counts did. The trust boundary is unchanged (same destination, same credential, and an agent could already send arbitrary OTLP); what changes is how much detail leaves by default. Worth spelling out, because two different hops are in play and only one of them carries a credential: - **agent→collector** (this section's hop) stays open on the bridge. Nothing downstream can tell one agent's export from another's. - **hive→swarm** authenticates. The swarm tier gives each hive its own receiver and stamps `hive=` from whichever one accepted the sample, so _that_ label is unforgeable — see [Authenticated ingest](#authenticated-ingest) below. A verified `hive` is reachable and a verified `agent` isn't — and that falls out of the topology rather than being a gap someone forgot to close. The swarm runs one collector, and the mechanism gives it no finer grain: a bearer-token check never reveals _which_ token matched, and a receiver reads request metadata rather than the claims it authenticated with. If you need per-agent numbers you can act on, take them from the agent's own turn-stats rather than from a metric label. ## Options reference The nix module (`nix/host-modules/otel.nix`) generates every `services.hyperhive.otel.*` option's full type/default/description/ example straight into [`/options/`](/options/) (host options — `nix build .#docs-host` for a local render). The build keeps that page honest in a way a hand-copied version here can't be, so it's the reference, not this doc. What follows is what a flat per-option listing can't express: the two-tier architecture, the security model, and how the options interact. ## The two collectors Telemetry crosses two collectors, and which one you configure depends on what the host is: | | runs where | receives from | does | | ------------------------------------ | ---------------------- | --------------------------------- | --------------------------------------------------------------------- | | **hive tier** — `otel.enable` | every hive with agents | that hive's agents, on the bridge | forwards to the swarm tier. Holds no credential, picks no destination | | **swarm tier** — `deploy.swarm-otel` | once per swarm | every hive's collector | writes the swarm's store and exports upstream | An all-local host runs both, and needs nothing said about the hop between them. ```nix services.hyperhive.otel = { enable = true; endpoint = "https://collector.example.com/otel"; # the upstream headersCredential = "/run/secrets/otel-headers"; # only the swarm tier reads it }; ``` **Why the hive tier isn't optional.** Exporting straight to `endpoint` means every agent needs the credential to authenticate — and the harness delivers that token into the agent's own `~/.claude/settings.json`, a file the agent can read. `0600` protects it from other containers, not from the agent itself. As long as the direct path stays _selectable_, that hole stays selectable; an option that can reintroduce it's a hole with extra steps. **Why the tiers stay separate on one box.** They're not collapsed when co-located: an all-local hive is a statement about _where_ processes run, not about the shape of the deployment. A boundary that disappears locally is one the local deployment stops testing. **`endpoint` keeps meaning "where telemetry goes upstream."** Neither tier redefines it — the agent-facing value is _derived_ (`http://:`), so an existing deployment's `endpoint` keeps working unchanged. The otel module contributes the bridge port to `exposeHostPorts` automatically; there is nothing to open by hand. ### Authenticated ingest The swarm tier gives **each hive its own receiver**, and stamps the `hive` label from whichever receiver accepted a sample. A hive therefore can't report metrics as another hive, and can't relabel its own by editing what it sends — the label isn't taken from the payload at all. **On an all-local swarm there is nothing to set.** Each hive already has an identity, and its collector reads the secret that host's own authelia minted. **On a hive that doesn't host the swarm's services**, the secret has to arrive somehow — copy it across and name it: ```nix services.hyperhive.otel.clientSecretFile = "/run/secrets/hive-telemetry.secret"; ``` **No unauthenticated mode exists.** A hive always presents an identity, so a missing credential is a build error rather than a quieter fallback — the collector has no anonymous route to accept samples on, and every path it serves belongs to exactly one hive. Getting the secret wrong shows up as the hive's collector logging 401s from the swarm tier and no metrics appearing for that hive. ⚠️ **`endpoint` must be valid for `protocol`.** The upstream exporter follows `otel.protocol` (`grpc` → the gRPC exporter, otherwise OTLP/HTTP), and the gRPC exporter takes an _address_: `https://host/path` is a legal `OTEL_EXPORTER_OTLP_ENDPOINT` for HTTP but fails as gRPC with _"missing port in address"_. nix validates the collector's config at build time, so a mismatch is a build error naming the reason rather than telemetry silently going nowhere. ## Network access Agent containers can only reach the host on ports 80 and 443 by default. To let them reach some other host-local service you run yourself — a database, a scratch HTTP endpoint — open its port on the bridge: ```nix services.hyperhive.network.exposeHostPorts = [ 5432 ]; ``` and point whatever consumes it at `10.42.0.1:5432` rather than loopback: inside a container, loopback is the _container_. The bridge IP is the host's address on the `hive-br0` bridge. The service must also bind an address the bridge can reach — a `127.0.0.1`-only listener stays unreachable no matter what the firewall allows. See `docs/networking/network.md::Reaching host services` for details. ⚠️ **None of this is needed for hyperhive's own telemetry** — `otel.enable` contributes the collector's port and derives the agent-facing endpoint itself. ## Built-in resource labels The harness sets the OTLP variables (`OTEL_EXPORTER_OTLP_ENDPOINT`, `_PROTOCOL`, `OTEL_RESOURCE_ATTRIBUTES`, the temporality preference) **container wide** — in systemd's `DefaultEnvironment` and in `/etc/profile` — so every process in an agent container exports to the hive's collector without any per-tool wiring. That covers Claude Code, `hive-metric`, and anything you run yourself from a tool call or a shell you open in the container. Every agent's export therefore includes these resource attributes automatically: | Attribute | Value | | -------------- | ------------------------------------------------------------ | | `service.name` | `hyperhive-agent` (constant) | | `agent` | agent logical name (for example `iris`) | | `hive` | hive display name (`services.hyperhive.hiveName`) | | `swarm` | swarm display name (`services.hyperhive.swarm.name`, if set) | Append additional labels via `extraResourceAttributes` (see option reference above); pass custom per-data-point labels with `hive-metric --labels` (see below). ## Log severity Every journald receiver in the pipeline maps the journal's `PRIORITY` onto an OpenTelemetry severity, so a line arrives in the log store already carrying its level and `severity_text:ERROR` is a query you can write: | `PRIORITY` | syslog | severity | | ---------- | ------------------ | -------- | | 0, 1, 2 | emerg, alert, crit | `FATAL` | | 3 | err | `ERROR` | | 4 | warning | `WARN` | | 5, 6 | notice, info | `INFO` | | 7 | debug | `DEBUG` | Both tiers that read a journal do this — the agent container's forwarder over its own journal, and the swarm collector over the host's — and both import the same mapping from `nix/journald-severity.nix`, which holds the table once. Note the direction. Syslog's priority counts **down** in urgency, where OpenTelemetry's severity number counts **up**. Neither is a passthrough of the other, and a pipeline that hands one straight to the other renders debug lines as critical without erroring anywhere. A record that reaches the store from somewhere other than a journal has no priority to map. Claude Code's own telemetry is the live example: it exports OTLP log records directly (`OTEL_LOGS_EXPORTER`, see `nix/agent-modules/claude-settings.nix`) and sets no severity on them, so they store as `Unspecified`. The **Log rows with no severity** panel on the `hyperhive · logs` dashboard counts both cases and keeps them apart: lines that had a priority and arrived without a severity anyway signal a broken mapping, while lines that never had one signal nothing of the sort. ### Why Explore's level buttons need the datasource told The stored field is `severity_text`, and Grafana's log-level buttons filter on a field called `level` — a name no row here carries. That's not something the pipeline can fix: VictoriaLogs' OTLP ingester names the stored field `severity_text` unconditionally (v1.52.0, `app/vlinsert/opentelemetry/pb.go`), and its ingest parameters have no `_level_field` sibling to rename it with. The datasource is told instead. `nix/host-modules/swarm-grafana.nix` provisions the VictoriaLogs datasource with a `logLevelRules` entry per severity in the table above, each matching `severity_text` against one of those values — which is the plugin's way of saying "the level lives in this field." `logLevelRules` is the datasource plugin's own jsonData key and the only level-related one it has — there is no single "the level lives in field X" string to set, so mapping every level takes one rule each. A level button then filters on `level` **or** the matching `severity_text`, and the store answers the second half. Each enabled rule appends an `OR :""` term to the query a level button emits, beside the `level:…` term that matches nothing here. Clicking "info" goes from ``` level:contains_common_case("info","information","informational","notice") ``` to that OR `severity_text:="INFO"`, which our rows do match. `Unspecified` is deliberately left out of those rules. It's VictoriaLogs' own rendering of an absent severity, and it's what the **Log rows with no severity** panel counts — giving it a level would dress the missing data up as a colour and retire the instrument that measures it. #### Three ways to get a `logLevelRules` entry wrong, all of them silent Grafana accepts any jsonData it doesn't recognise, so a bad rule provisions cleanly and the affected level button goes on returning zero rows: - `enabled` must be literally `true`, not merely not-false. The query builder keeps rules on `rule.enabled` being truthy while the row colouring path keeps them on `!== false`, so an omitted flag colours rows correctly and leaves the buttons broken — working in the half nobody is looking at. - `level` must be a canonical Grafana `LogLevel` value. The builder groups rules by it and only ever looks up `critical`, `error`, `warning`, `info`, `debug`, `trace`; `warn` and `fatal` are enum aliases that resolve to other spellings and match no group. - `value` is compared with `===`, so it must be the severity text exactly as stored: the uppercase OpenTelemetry short names `overwrite_text` writes in `nix/journald-severity.nix`, not that table's lowercase keys. ## Host-emitted container-resource metrics (hive-c0re) When OTEL is enabled, **hive-c0re itself** also exports each agent container's resource load — the same cgroup gauges shown on the dashboard LOAD tab — to this hive's own collector, exactly like an agent does and with no separate toggle. These come from the host, not the in-container Claude SDK, so they cover containers even when their agent is idle. Emitted via the OpenTelemetry Rust SDK, using the [semconv `container.*`](https://opentelemetry.io/docs/specs/semconv/system/container-metrics/) metric names + the standard `container.name` attribute where a spec metric exists, so off-the-shelf OTEL/Grafana container dashboards work. Resource `service.name = hyperhive-c0re`; hive-c0re tags each data point `container.name` (= the `h-` machine) and the hive `agent` label: | Metric | Unit | Kind | Source | | ----------------------------------- | ---- | ------- | ----------------------------------------------------------------------------------------------------------- | | `container.cpu.time` | `s` | counter | cumulative `cpu.stat` `usage_usec` → seconds | | `container.memory.usage` | `By` | gauge | `memory.current` | | `hyperhive.container.memory.limit` | `By` | gauge | `memory.max` (custom — semconv has no `.limit` metric; omitted when unlimited) | | `hyperhive.container.memory.peak` | `By` | gauge | `memory.peak` (custom — no semconv metric; omitted if unavailable) | | `hyperhive.container.storage.usage` | `By` | gauge | state dir + writable rootfs (custom — semconv only has `disk.io`; omitted until the slow disk sampler runs) | | `hyperhive.container.cpu.percent` | `%` | gauge | host-normalised percent (custom — the value the dashboard LOAD tab shows, no `rate()` needed) | The `hyperhive.`-prefixed metrics have no semconv equivalent (memory limit + peak, on-disk footprint, and an instantaneous cpu percent kept alongside the spec `container.cpu.time` counter for convenience). Hive labels (`hive`, `swarm`, …) ride on the resource via `extraResourceAttributes`. Cadence follows `metricIntervalMs` (default 60s). Transport is OTLP/HTTP (JSON) to the hive collector's bridge address, with no auth header — that first hop is unauthenticated for every producer on this host, and the upstream credential stays on the swarm tier. ## Agent-emitted per-turn metrics (`hive-agent`) When OTEL is enabled, the harness itself (`hive-agent`) exports one small set of metrics per claude turn, recorded the moment the turn ends (not polled). These are deliberately the fields Claude Code's own built-in export (see above) can't know about — the harness's own wall-clock timing, what woke the turn, its own outcome classification, the loose-ends backlog, and session boundaries. Token usage, cost, and tool-call counts are **not** duplicated here; that's already covered by Claude's own export. | Metric | Unit | Kind | Attributes | | -------------------------------------- | ---- | --------- | ----------------------------------------------------------------------------------------------------------- | | `hyperhive.agent.turn.duration` | `ms` | histogram | `wake_from`, `result_kind`, `model` | | `hyperhive.agent.turn.count` | — | counter | `wake_from`, `result_kind`, `model` | | `hyperhive.agent.session.count` | — | counter | `model` (incremented once per fresh, non-`--continue`'d session) | | `hyperhive.agent.loose_ends.threads` | — | gauge | none | | `hyperhive.agent.loose_ends.reminders` | — | gauge | none | | `hyperhive.agent.claude_md.lines` | — | gauge | none — recorded from the `CLAUDE.md`-size watch's own ~15-minute tick, **not** per turn like the rows above | Resource attributes (`service.name`, `agent`, `hive`, `swarm`) come from the same container-wide `OTEL_RESOURCE_ATTRIBUTES` as everything else in this section — nothing extra to configure. Cadence follows `HYPERHIVE_OTEL_METRIC_INTERVAL_MS` (default 60s, same variable + default as `hive-c0re`'s container-resource export above) — that only controls how often the harness flushes the batched points to the collector, not how often it records them (every turn, always). ## Hive-scoped metrics (hive-c0re) hive-c0re measures everything above **per agent**, tagged with the hive it runs in. It measures these three per **hive** instead, and they carry no `agent` label — so a hive that hosts no agents still reports, and "this hive is quiet" is distinguishable from "this hive is gone." Select them with `{hive!="",agent=""}`. | Metric | Unit | Kind | Meaning | | ------------------------- | ---- | ----- | -------------------------------------------------------------------------------------------------------------------- | | `process.uptime` | `s` | gauge | seconds since this hive's `hive-c0re` started exporting; a restart reads as a drop to ~0 | | `hyperhive.hive.degraded` | `1` | gauge | `1` while the hive reports itself unhealthy — the same verdict `/health/ready` gives and the swarm status view shows | | `hyperhive.hive.warnings` | `1` | gauge | how many warnings are currently raised, split by a `level` attribute (`warn`, `crit`) | hive-c0re reports both levels every cycle, `0` included, so a healthy hive is visible as zeros rather than as missing series. `hyperhive.hive.degraded` is what a dashboard should alert on: it's `hive-c0re`'s own readiness verdict, so it stays in step with `/health/ready` and with what the swarm controller sees. `hyperhive.hive.warnings` is the detail behind it — `warn`-level entries mean "an operator should look" and do **not** set `degraded`. Same cadence, transport and resource labels as the container metrics above. ## VCS activity metrics (`swarm-controller`) `swarm-controller` registers a single instance-wide Forgejo webhook (a "global/system" hook, not scoped to any one org or repo) and counts commit and push activity as deliveries arrive — occurrence-driven, not polled. Forgejo's own native `/metrics` endpoint has no equivalent: it exposes counts of durable rows (issues, comments, repos), and Forgejo doesn't store either a commit or a push anywhere as a row to count. | Metric | Unit | Kind | Attributes | | ---------------------------- | ---- | ------- | ------------------- | | `hyperhive.vcs.commit.count` | — | counter | `repo` (`org/repo`) | | `hyperhive.vcs.push.count` | — | counter | `repo` (`org/repo`) | A push with zero commits (a branch delete, or a force-push that doesn't add new commits) still increments `push.count`; `commit.count` only advances when the delivery actually carries commits. Same enable signal (`OTEL_EXPORTER_OTLP_ENDPOINT`), cadence variable (`HYPERHIVE_OTEL_METRIC_INTERVAL_MS`) and `HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES` resource-attribute channel as `swarm-controller`'s other OTEL exporter (its `hive-jobq-metrics`-backed job-graph rollup, undocumented here — see that crate's own doc comment) — `swarm-controller` sets `service.name = swarm-controller` directly rather than reading it from the container environment, since it's a standalone daemon, not a per-agent harness process. ## Agent-emitted custom metrics (`hive-metric`) Agents can push arbitrary labeled metrics to the same OTEL collector via the `hive-metric` CLI tool, available in every agent container when `services.hyperhive.otel.enable = true`. ### Usage ```text hive-metric [--type counter|gauge] [--temporality delta|cumulative] [--labels key=value...] ``` - `` — metric name (for example `tasks_completed`, `latency_ms`). - `` — numeric value (f64; integers and floats both accepted). - `--type counter|gauge` — metric kind: `counter` (increasing sum, default) or `gauge` (instantaneous point-in-time value). - `--temporality delta|cumulative` — counter reporting mode (`counter` only, ignored for `gauge`): `delta` (this call's own contribution, default — send `1` each time and the collector accumulates) or `cumulative` (this call reports the running total, which a stateless one-shot CLI can't track itself). - `--labels key=value` — extra per-data-point labels. Repeat the flag for more than one. `hive-metric` inherits the resource labels (agent, hive, swarm, service.name) automatically from `OTEL_RESOURCE_ATTRIBUTES` — don't re-specify them. ### Examples ```text # Counter: one more task finished (delta is the default — no flag needed) hive-metric tasks_completed 1 --labels phase=scan # Gauge: current queue depth (absolute value — must use --type gauge) hive-metric queue_depth 17 --type gauge # Float gauge with multiple labels (instantaneous measurement) hive-metric api_latency_ms 142.5 --type gauge --labels model=sonnet --labels tier=api ``` ### Error when OTEL isn't configured When `services.hyperhive.otel.enable = false` (the default), the `OTEL_EXPORTER_OTLP_ENDPOINT` env var isn't set and `hive-metric` exits with an informative error message. No silently dropped metrics. ### Wire format `hive-metric` always uses **OTLP HTTP/JSON** (`application/json` POST to `$OTEL_EXPORTER_OTLP_ENDPOINT/v1/metrics`), regardless of the `OTEL_EXPORTER_OTLP_PROTOCOL` setting. `hive-metric` forwards auth headers from `OTEL_EXPORTER_OTLP_HEADERS` verbatim. ## Metrics temporality The harness configures OTEL export with **cumulative** temporality by default (`OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=cumulative`), overriding Claude Code's default of DELTA. This avoids silent metric drops in Prometheus-family backends (including Grafana LGTM / Mimir) that don't ship a delta-to-cumulative processor. **`hive-metric` counters are the one exception**, reporting delta by default (see above) — programmatically set on the exporter, which overrides this container-wide env var for that tool specifically. `--type gauge` is unaffected either way; gauges have no temporality. The hive-tier collector runs a `deltatocumulative` processor ahead of export, so a delta `hive-metric` counter still lands in VictoriaMetrics as a cumulative series — the standard `rate()`/`increase()` idioms work on it exactly like any other counter in this system, no special query needed.