hyperhive/docs/scheduler/observability.md

Observability (OpenTelemetry)

The swarm has one telemetry pipeline. The swarm's collector writes the swarm's metrics and log stores and, if you name one, exports upstream; each hive runs a collector of its own that forwards its agents' telemetry there. What flows through it: each agent's Claude Code statistics (token usage, cost, tool call counts) via Claude Code's built-in OpenTelemetry integration, each agent container's journal, the hive tier's own hive-c0re.service journal line for a detected container crash (see Container-crash alerting), and the hyperhive metrics catalogued below.

The two collectors

Telemetry crosses two collectors, and which one you configure depends on what the host is:

	runs where	receives from	does
swarm tier — deploy.swarm-otel	once per swarm	every hive's collector	writes the swarm's stores and exports upstream
hive tier — otel.enable	every hive with agents	that hive's agents, on the bridge	forwards to the swarm tier. Holds no credential

An all-local host runs both; the hop between them configures itself.

services.hyperhive.otel = {
  enable            = true;
  endpoint          = "https://collector.example.com/otel";  # the upstream
  headersCredential = "/run/secrets/otel-headers";           # only the swarm tier reads it
};

Enabling export

otel.enable is the single gate on a hive: one switch in the host config covers every agent container on it. endpoint is where telemetry ends up after it leaves the swarm — optional, because the swarm's own metrics store (deploy.victoriametrics) is a destination in its own right. With both, telemetry goes to both. See swarm/services.md.

Telemetry leaves a hive exactly one way: through the collector that enable starts on the host. Agents never talk to endpoint themselves — they export unauthenticated to a collector address only their own containers can reach. They dial it as http://hive-collector.<swarm domain>:<collector port>, a name identical on every hive, which each hive's own dnsmasq answers with its bridge IP, so an agent's build carries no hive-specific address. The otel module opens that port on the bridge itself. That collector forwards to the swarm's (swarm/services.md), the single process holding the upstream credential and the only writer to the swarm's stores. No agent holds a copy, and neither does the hive.

The hive collector reaches the swarm collector by its gateway name (swarm.otel.domain, default otel.<swarm domain>) — the same DNS-and-CA-trust shape every hive-to-swarm-service hop uses. On the host running the swarm collector the hive's dnsmasq answers that name; elsewhere it resolves through ordinary DNS.

⚠️ The hive collector carries every bit of that hive's telemetry. It runs on the same host as the agents and restarts on failure. Telemetry isn't the control plane, so degraded telemetry isn't degraded operation.

Why two tiers

Agents export only to the hive tier. Exporting straight to endpoint would mean every agent needs the credential — and the only place to hand it to an agent container is somewhere the agent itself can read, its own claude settings among them. 0600 protects a secret from other containers, not from the agent it belongs to.

The tiers stay separate on one box. An all-local hive is a statement about where processes run, not about the shape of the deployment. A boundary that disappears locally is one the local deployment stops testing.

What the agent→collector hop is and isn't

It has no application-level auth. The receiver takes any OTLP that reaches it; what bounds who can reach it's the firewall — exposeHostPorts opens the port on the bridge interface only — "unauthenticated to a bridge address" means reachable from an agent container, not presents a credential.

The consequence, stated because it's a choice rather than an oversight: any agent can push arbitrary OTLP, and it's forwarded on under the operator's credential. Neither tier can tell a container's genuine Claude Code stats from anything else shaped like OTLP arriving on that port — including data smuggled out in resource attributes on an otherwise-legitimate export.

That's a different risk from the one the collector fixes: an agent that held the upstream credential could do everything above and use the token anywhere else. The collector removes the token and keeps the pipe. Agents are inside the trust boundary (docs/trust-boundary/security.md: capability = accepted risk), so an agent being able to send is an accepted extension of that boundary — but it's not closed by this design, and don't read anything here as closing it.

The agent label is self-reported, and the hive→swarm authentication below doesn't change that. Treat it as a convenience for grouping dashboards, never as evidence of which container produced a sample: any agent that can reach this hive's collector can label its data as any other agent.

Logs ride the same hop, and a journal carries more than a counter does. Each agent container forwards its own journal through this port — every unit in it at info and above, not an allowlist. That's the harness, the MCP daemons and whatever a tool call spawned, so command lines and error text leave the container, not just counts. Destination, credential and trust boundary are the same as for any OTLP the agent sends.

Worth spelling out, because two different hops are in play and only one of them carries a credential:

• agent→collector (this section's hop) stays open on the bridge. Nothing downstream can tell one agent's export from another's.
• hive→swarm authenticates. The swarm tier gives each hive its own receiver and stamps hive= from whichever one accepted the sample, so that label is unforgeable — see Authenticated ingest below.

A verified hive is reachable and a verified agent isn't — and that falls out of the topology rather than being a gap someone forgot to close. The swarm runs one collector, and the mechanism gives it no finer grain: a bearer-token check never reveals which token matched, and a receiver reads request metadata rather than the claims it authenticated with.

If you need per-agent numbers you can act on, take them from the agent's own turn-stats rather than from a metric label.

Authenticated ingest

The swarm tier gives each hive its own receiver, and stamps the hive label from whichever receiver accepted a sample. A hive therefore can't report metrics as another hive, and can't relabel its own by editing what it sends — the label isn't taken from the payload at all.

Nothing to set where the hive holds its store identity. The hive's collector reads its secret from the swarm's secret store under the hive's own leaf (deploy.bao.clientCertFile): swarm-bao-hive-otel-oidc.service fetches swarm/hives/<hive>/oidc/client, which the publisher in authelia's container writes. It runs at boot and retries every 30s until the store answers; when a run lands a changed value it restarts the collector. A timer runs it again every ten minutes, so a secret rotated later reaches the collector within about ten minutes. On a hive that runs the store the leaf is already there; elsewhere it's the credential you place — see secret placement.

To deliver the secret some other way, name the file. The fetch unit isn't rendered, and the collector waits up to 120s for the file to appear:

services.hyperhive.otel.clientSecretFile = "/run/secrets/hive-telemetry.secret";

No unauthenticated mode exists. A hive always presents an identity, so a missing credential is a build error rather than a quieter fallback — the collector has no anonymous route to accept samples on, and every path it serves belongs to exactly one hive.

Getting the secret wrong shows up as the hive's collector logging 401s from the swarm tier and no metrics appearing for that hive.

⚠️ endpoint must be valid for protocol. The upstream exporter follows otel.protocol (grpc → the gRPC exporter, otherwise OTLP/HTTP), and the gRPC exporter takes an address: https://host/path is a legal OTEL_EXPORTER_OTLP_ENDPOINT for HTTP but fails as gRPC with "missing port in address". nix validates the collector's config at build time, so a mismatch is a build error naming the reason rather than telemetry silently going nowhere.

Network access

Agent containers reach the host only on ports 80 and 443 (plus DNS). The collector port needs nothing from you: otel.enable opens it on the bridge. To let agents reach some other host-local service of your own → networking/network.md.

Options reference

The nix module (nix/host-modules/otel.nix) generates every services.hyperhive.otel.* option's full type, default, description and example into the options reference (host options — nix build .#docs-host for a local render). The same module declares the collector port every hive and its agents share, services.hyperhive.swarm.hiveOtel.collectorPort, and each hive's own services.hyperhive.deploy.hive-controller.otel.extraResourceAttributes. That's the reference; this page covers what a per-option listing can't: the two-tier architecture, the security model, and how the options interact.

Built-in resource labels

The harness sets the OTLP variables (OTEL_EXPORTER_OTLP_ENDPOINT, _PROTOCOL, OTEL_RESOURCE_ATTRIBUTES, the temporality preference) container wide — in systemd's DefaultEnvironment and in /etc/profile — so every process in an agent container exports to the hive's collector without any per-tool wiring. That covers Claude Code, hive-metric, and anything you run yourself from a tool call or a shell you open in the container.

Every agent's export therefore includes these resource attributes automatically:

Attribute	Value
service.name	hyperhive-agent (constant)
agent	agent logical name (for example iris)
swarm	swarm display name (services.hyperhive.swarm.name, if set)

The swarm's collector adds hive, the name of the hive whose receiver accepted the push (see Authenticated ingest). The container doesn't set it.

Append additional labels for one hive via extraResourceAttributes (see option reference above); pass custom per-data-point labels with hive-metric --labels (see below).

Log severity

Every journald receiver in the pipeline maps the journal's PRIORITY onto an OpenTelemetry severity, so a line arrives in the log store already carrying its level and severity_text:ERROR is a query you can write:

PRIORITY	syslog	severity
0, 1, 2	emerg, alert, crit	FATAL
3	err	ERROR
4	warning	WARN
5, 6	notice, info	INFO
7	debug	DEBUG

Both tiers that read a journal do this — the agent container's forwarder over its own journal, and the swarm collector over the host's — and both import the same mapping from nix/journald-severity.nix, which holds the table once.

Note the direction. Syslog's priority counts down in urgency, where OpenTelemetry's severity number counts up. Neither is a passthrough of the other, and a pipeline that hands one straight to the other renders debug lines as critical without erroring anywhere.

A record that reaches the store from somewhere other than a journal has no priority to map. Claude Code's own telemetry is the live example: it exports OTLP log records directly (OTEL_LOGS_EXPORTER, see nix/agent-modules/claude-settings.nix) and sets no severity on them, so they store as Unspecified. The Log rows with no severity panel on the hyperhive · logs dashboard counts both cases and keeps them apart: lines that had a priority and arrived without a severity anyway signal a broken mapping, while lines that never had one signal nothing of the sort.

Why Explore's level buttons need the datasource told

The stored field is severity_text, and Grafana's log-level buttons filter on a field called level — a name no row here carries. That's not something the pipeline can fix: VictoriaLogs' OTLP ingester names the stored field severity_text unconditionally (v1.52.0, app/vlinsert/opentelemetry/pb.go), and its ingest parameters have no _level_field sibling to rename it with.

nix/host-modules/swarm-grafana.nix tells the datasource instead, provisioning it with a logLevelRules entry per severity in the table above, each matching severity_text against one of those values — which is the plugin's way of saying "the level lives in this field." logLevelRules is the datasource plugin's own jsonData key and the only level-related one it has — there is no single "the level lives in field X" string to set, so mapping every level takes one rule each. A level button then filters on level or the matching severity_text, and the store answers the second half. Each enabled rule appends an OR <field>:<op>"<value>" term to the query a level button emits, beside the level:… term that matches nothing here. Clicking "info" goes from

level:contains_common_case("info","information","informational","notice")

to that OR severity_text:="INFO", which our rows do match.

Unspecified is deliberately left out of those rules. It's VictoriaLogs' own rendering of an absent severity, and it's what the Log rows with no severity panel counts — giving it a level would dress the missing data up as a colour and retire the instrument that measures it.

Three ways to get a logLevelRules entry wrong, all of them silent

Grafana accepts any jsonData it doesn't recognise, so a bad rule provisions cleanly and the affected level button goes on returning zero rows:

• enabled must be literally true, not merely not-false. The query builder keeps rules on rule.enabled being truthy while the row colouring path keeps them on !== false, so an omitted flag colours rows correctly and leaves the buttons broken — working in the half nobody is looking at.
• level must be a canonical Grafana LogLevel value. The builder groups rules by it and only ever looks up critical, error, warning, info, debug, trace; warn and fatal are enum aliases that resolve to other spellings and match no group.
• the builder compares value with ===, so it must be the severity text exactly as stored: the uppercase OpenTelemetry short names overwrite_text writes in nix/journald-severity.nix, not that table's lowercase keys.

Host-emitted container-resource metrics (hive-c0re)

When OTEL is enabled, hive-c0re itself also exports each agent container's resource load — the same cgroup gauges — to this hive's own collector, exactly like an agent does and with no separate toggle. These come from the host, not the in-container Claude SDK, so they cover containers even when their agent is idle.

Emitted via the OpenTelemetry Rust SDK, using the semconv container.* metric names + the standard container.name attribute where a spec metric exists, so off-the-shelf OTEL/Grafana container dashboards work. Resource service.name = hyperhive-c0re; hive-c0re tags each data point container.name (= the h-<agent> machine) and the hive agent label:

Metric	Unit	Kind	Source
container.cpu.time	s	counter	cumulative cpu.stat usage_usec → seconds
container.memory.usage	By	gauge	memory.current
hyperhive.container.memory.limit	By	gauge	memory.max (custom — semconv has no .limit metric; omitted when unlimited)
hyperhive.container.memory.peak	By	gauge	memory.peak (custom — no semconv metric; omitted if unavailable)
hyperhive.container.storage.usage	By	gauge	state/ + work/ + writable rootfs, measured by hive-priv every 30 min (custom — semconv only has disk.io; omitted while unknown: not measured yet, or the last measurement failed)
hyperhive.container.cpu.percent	%	gauge	host-normalised percent (custom — no rate() needed)

The hyperhive.-prefixed metrics have no semconv equivalent (memory limit + peak, on-disk footprint, and an instantaneous cpu percent kept alongside the spec container.cpu.time counter for convenience). The swarm label rides on the resource, and the swarm collector adds hive from the receiver that accepted the sample. Any other label goes in the hive's extraResourceAttributes.

The export cadence is a fixed 60s. Transport is OTLP/HTTP (JSON) to the hive collector's bridge address, with no auth header — that first hop is unauthenticated for every producer on this host, and the upstream credential stays on the swarm tier.

Container-crash alerting (hive-c0re)

The crash watcher (crash_watch.rs, a 10s poll of every managed container) classifies a container that stopped without an operator-initiated action as a crash and tracing::warn!s "container crash detected" — nothing else: no broker event, no per-hive dashboard banner. Both of those used to exist; this change removed them in favour of the log line.

When otel.enable is on, the hive tier's own collector runs a journald receiver scoped to hive-c0re.service (no bind mount needed — unlike an agent's journal, this collector already shares a filesystem with the unit it reads, being a host systemd service rather than a container) and forwards that one log line alongside the rest of the hive's telemetry. A swarm-level grafana alert, hh-container-crashed in nix/host-modules/swarm-grafana.nix, matches it directly and pages the operator — no agent, and no per-hive dashboard, in the loop.

Requires services.journald.storage = "persistent" (the NixOS default) on every hive host with otel.enable set — asserted at eval time.

⚠️ A hive with otel.enable off has no operator-visible crash signal at all — the per-hive dashboard banner this replaces no longer exists, and there is no fallback path. Turning on otel.enable is what makes a crash reach anyone.

Agent-emitted per-turn metrics (hive-agent)

When OTEL is enabled, the harness itself (hive-agent) exports one small set of metrics per claude turn, recorded the moment the turn ends (not polled). These are deliberately the fields Claude Code's own built-in export (see above) can't know about — the harness's own wall-clock timing, what woke the turn, its own outcome classification, the loose-ends backlog, session boundaries, and how full the context window is. Token usage and tool-call counts are not duplicated here; that's already covered by Claude's own export. Cost is here only for agents on an ACP runtime, which have no export of their own — see ACP-reported cost.

Metric	Unit	Kind	Attributes
hyperhive.agent.turn.duration	ms	histogram	wake_from, result_kind, model
hyperhive.agent.turn.count	—	counter	wake_from, result_kind, model
hyperhive.agent.session.count	—	counter	model (incremented once per fresh, non---continue'd session)
hyperhive.agent.loose_ends.threads	—	gauge	none
hyperhive.agent.loose_ends.reminders	—	gauge	none
hyperhive.agent.claude_md.lines	—	gauge	none — recorded from the AGENTS.md-size watch's own ~15-minute tick, not per turn like the rows above
hyperhive.agent.subagent_role.lines	—	gauge	role (the file stem) — one series per role file in subagent_roles/, on the same ~15-minute tick
hyperhive.agent.claude_usage.percent	%	gauge	window (five_hour, seven_day, … as the usage endpoint names them) — polled every 5 minutes, not per turn
hyperhive.agent.claude_usage.resets_at	s	gauge	window — unix seconds at which that window resets; same 5-minute poll
hyperhive.agent.context.used	{token}	gauge	none — tokens in the context window at turn end, the numerator of the web UI's ctx%
hyperhive.agent.context.size	{token}	gauge	none — the context window those tokens fill, ctx%'s denominator
hyperhive.agent.cost.usage	—	counter	model, currency — ACP agents only, see below

An agent with no subagent_roles/ directory exports no subagent_role.lines series. A role file deleted while the harness runs keeps its last value until the harness restarts.

For the two claude_usage gauges the harness polls the Claude subscription usage endpoint (GET /api/oauth/usage, the one behind claude's own /usage) with the OAuth access token from the agent's own ~/.claude/.credentials.json. The harness only reads that token and never refreshes it — claude owns refresh-token rotation. An agent with no OAuth session (API-key backends, not yet logged in) or an expired token skips the poll, so a gauge keeps its last value until the next successful poll — a resets_at in the past means the paired percent is stale.

The two context gauges hold the two numbers the web UI divides for its ctx%: the last turn's context tokens (input, cache read and cache creation), and the API-reported window, else the model's default. A turn that parsed no usage leaves both where the previous turn put them. The dashboard's Context window used by agent panel (health tab) divides one by the other.

Resource attributes (service.name, agent, swarm) come from the same container-wide OTEL_RESOURCE_ATTRIBUTES as everything else in this section, and the swarm's collector adds hive — nothing extra to configure. The export cadence is a fixed 60s, the same as hive-c0re's container-resource export above. That only controls how often the harness flushes the batched points to the collector, not how often it records them (every turn, always).

ACP-reported cost

An ACP agent reports what its session has cost so far in the cost field of its usage_update notifications — a running total, not a per-turn figure (opencode sums every assistant message in the session). The harness counts what each report adds to the last one and exports that as hyperhive.agent.cost.usage, so increase() over it gives the money spent in a range. It follows a few rules:

• A new session's first report counts in full, since the session started at zero.
• A session loaded into a freshly started agent (after a harness restart or an agent crash) has an unknown total until it reports one, so that first report only sets the baseline — the turn it covers goes uncounted.
• A total lower than the previous one means the agent dropped part of the session's history; it adds nothing and becomes the new baseline.
• Cost from a compaction the harness runs between turns counts toward the next turn.

The value is in whatever currency the agent names; opencode always sends USD, and the hyperhive · agents dashboard's ACP cost panels (cost tab) filter on it. A claude agent never records this metric: claude's own export already carries its cost as claude_code.cost.usage, and keeping the two apart means no turn is ever counted twice. Add both names together for a whole-swarm spend figure.

Hive-scoped metrics (hive-c0re)

hive-c0re measures everything above per agent, tagged with the hive it runs in. It measures these three per hive instead, and they carry no agent label — so a hive that hosts no agents still reports, and "this hive is quiet" is distinguishable from "this hive is gone." Select them with {hive!="",agent=""}.

Metric	Unit	Kind	Meaning
process.uptime	s	gauge	seconds since this hive's hive-c0re started exporting; a restart reads as a drop to ~0
hyperhive.hive.degraded	1	gauge	1 while the hive reports itself unhealthy — the same verdict /health/ready gives and the swarm status view shows
hyperhive.hive.warnings	1	gauge	how many warnings are currently raised, split by a level attribute (warn, crit)

hive-c0re reports both levels every cycle, 0 included, so a healthy hive is visible as zeros rather than as missing series.

hyperhive.hive.degraded is what a dashboard should alert on: it's hive-c0re's own readiness verdict, so it stays in step with /health/ready and with what the swarm controller sees. hyperhive.hive.warnings is the detail behind it — warn-level entries mean "an operator should look" and do not set degraded.

Same cadence, transport and resource labels as the container metrics above.

VCS activity metrics (swarm-controller)

swarm-controller registers a single instance-wide Forgejo webhook (a "global/system" hook, not scoped to any one org or repo) and counts commit and push activity as deliveries arrive — occurrence-driven, not polled. Forgejo's own native /metrics endpoint has no equivalent: it exposes counts of durable rows (issues, comments, repos), and Forgejo doesn't store either a commit or a push anywhere as a row to count.

Metric	Unit	Kind	Attributes
hyperhive.vcs.commit.count	—	counter	repo (org/repo)
hyperhive.vcs.push.count	—	counter	repo (org/repo)

A push with zero commits (a branch delete, or a force-push that doesn't add new commits) still increments push.count; commit.count only advances when the delivery actually carries commits. Same enable signal (OTEL_EXPORTER_OTLP_ENDPOINT), fixed 60s cadence and HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES resource-attribute channel as swarm-controller's other OTEL exporter (its hive-jobq-metrics-backed job-graph rollup, undocumented here — see that crate's own doc comment) — swarm-controller sets service.name = swarm-controller directly rather than reading it from the container environment, since it's a standalone daemon, not a per-agent harness process.

Both exporters push through swarm-controller's own authenticated client, and the collector's oidc/swarm receiver checks the bearer. When a push fails, swarm-controller logs otel export: push to the collector failed at warn, with the kind of failure (token, transport or status <code>) and the start of the response body. It repeats a persisting failure every 10 minutes with a count, and logs once at info when pushes succeed again.

Agent-emitted custom metrics (hive-metric)

Agents can push arbitrary labeled metrics to the same OTEL collector via the hive-metric CLI tool, available in every agent container when services.hyperhive.otel.enable = true.

Usage

hive-metric <name> <value> [--type counter|gauge] [--temporality delta|cumulative] [--labels key=value...]

• <name> — metric name (for example tasks_completed, latency_ms).
• <value> — numeric value (f64; integers and floats both accepted).
• --type counter|gauge — metric kind: counter (increasing sum, default) or gauge (instantaneous point-in-time value).
• --temporality delta|cumulative — counter reporting mode (counter only, ignored for gauge): delta (this call's own contribution, default — send 1 each time and the collector accumulates) or cumulative (this call reports the running total, which a stateless one-shot CLI can't track itself).
• --labels key=value — extra per-data-point labels. Repeat the flag for more than one. hive-metric inherits the resource labels (agent, swarm, service.name) automatically from OTEL_RESOURCE_ATTRIBUTES — don't re-specify them.

Examples

# Counter: one more task finished (delta is the default — no flag needed)
hive-metric tasks_completed 1 --labels phase=scan

# Gauge: current queue depth (absolute value — must use --type gauge)
hive-metric queue_depth 17 --type gauge

# Float gauge with multiple labels (instantaneous measurement)
hive-metric api_latency_ms 142.5 --type gauge --labels model=sonnet --labels tier=api

Error when OTEL isn't configured

When services.hyperhive.otel.enable = false (the default), the OTEL_EXPORTER_OTLP_ENDPOINT env var isn't set and hive-metric exits with an informative error message. No silently dropped metrics.

Wire format

hive-metric always uses OTLP HTTP/JSON (application/json POST to $OTEL_EXPORTER_OTLP_ENDPOINT/v1/metrics), regardless of the OTEL_EXPORTER_OTLP_PROTOCOL setting. hive-metric forwards auth headers from OTEL_EXPORTER_OTLP_HEADERS verbatim.

Metrics temporality

The harness configures OTEL export with cumulative temporality by default (OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=cumulative), overriding Claude Code's default of DELTA. This avoids silent metric drops in Prometheus-family backends (including Grafana LGTM / Mimir) that don't ship a delta-to-cumulative processor.

hive-metric counters are the one exception, reporting delta by default (see above) — programmatically set on the exporter, which overrides this container-wide env var for that tool specifically. --type gauge is unaffected either way; gauges have no temporality. The hive-tier collector runs a deltatocumulative processor ahead of export, so a delta hive-metric counter still lands in VictoriaMetrics as a cumulative series — the standard rate()/increase() idioms work on it exactly like any other counter in this system, no special query needed.