vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

every hive runs its otel collector, and agents export to it by default

Browse source
This commit is contained in:
atlas 2026-10-18 23:17:28 +02:00 • committed by mara
commit 13daf4c7dc
28 changed files with 245 additions and 303 deletions
Download patch file Download diff file Expand all files Collapse all files

49
docs/scheduler/observability.md
View file

@ -15,16 +15,15 @@ hyperhive metrics catalogued below.
Telemetry crosses two collectors, and which one you configure depends on what
the host is:
| | runs where | receives from | does |
| ------------------------------------ | ---------------------- | --------------------------------- | ----------------------------------------------- |
| **swarm tier** — `deploy.swarm-otel` | once per swarm | every hive's collector | writes the swarm's stores and exports upstream |
| **hive tier** — `otel.enable` | every hive with agents | that hive's agents, on the bridge | forwards to the swarm tier. Holds no credential |
| | runs where | receives from | does |
| ------------------------------------ | -------------- | --------------------------------- | ----------------------------------------------- |
| **swarm tier** — `deploy.swarm-otel` | once per swarm | every hive's collector | writes the swarm's stores and exports upstream |
| **hive tier** — always on | every hive | that hive's agents, on the bridge | forwards to the swarm tier. Holds no credential |
An all-local host runs both; the hop between them configures itself.
```nix
services.hyperhive.otel = {
enable = true;
endpoint = "https://collector.example.com/otel"; # the upstream
headersCredential = "/run/secrets/otel-headers"; # only the swarm tier reads it
};
@ -32,20 +31,20 @@ services.hyperhive.otel = {
## Enabling export
`otel.enable` is the single gate on a hive: one switch in the host config
covers every agent container on it.
Every hive runs its collector and every agent exports to it, with nothing to
switch on: there is no per-hive or per-agent opt-in.
`endpoint` is where telemetry ends up after it leaves the swarm — optional,
because the swarm's own metrics store (`deploy.victoriametrics`) is a
destination in its own right. With both, telemetry goes to both. See
[`swarm/services.md`](../swarm/services.md#metrics-victoriametrics--grafana).
**Telemetry leaves a hive exactly one way: through the collector that
`enable` starts on the host.** Agents never talk to `endpoint` themselves —
runs on the host.** Agents never talk to `endpoint` themselves —
they export unauthenticated to a collector address only their own containers
can reach. They dial it as
`http://hive-collector.<swarm domain>:<collector port>`, a name identical on
every hive, which each hive's own dnsmasq answers with its bridge IP, so an
agent's build carries no hive-specific address. The otel module opens that
`http://hive-collector.hive:4318`, a name identical on
every hive and every swarm, which each hive's own dnsmasq answers with its bridge IP, so an
agent's build carries no hive-specific address and no swarm domain. The otel module opens that
port on the bridge itself. That collector forwards to the swarm's
([`swarm/services.md`](../swarm/services.md#telemetry-collector-otel)), the
single process holding the upstream credential and the only writer to the
@ -170,7 +169,7 @@ a build error naming the reason rather than telemetry silently going nowhere.
## Network access
Agent containers reach the host only on ports 80 and 443 (plus DNS). The
collector port needs nothing from you: `otel.enable` opens it on the bridge.
collector port needs nothing from you: the hive collector opens it on the bridge.
To let agents reach some other host-local service of your own →
[`networking/network.md`](../networking/network.md#reaching-host-services-exposehostports).
@ -180,7 +179,7 @@ The nix module (`nix/host-modules/otel.nix`) generates every
`services.hyperhive.otel.*` option's full type, default, description and
example into the [options reference](https://hyperhive.darkest.space/options/)
(host options — `nix build .#docs-host` for a local render). The same module
declares the collector port every hive and its agents share,
fixes the collector port every hive and its agents share,
`services.hyperhive.swarm.hiveOtel.collectorPort`, and each hive's own
`services.hyperhive.deploy.hive-controller.otel.extraResourceAttributes`.
That's the reference; this page covers what a per-option listing can't: the
@ -296,7 +295,7 @@ cleanly and the affected level button goes on returning zero rows:
<!-- vale write-good.Passive = NO -->
When OTEL is enabled, **hive-c0re itself** also exports each agent
**hive-c0re itself** also exports each agent
container's resource load — the same cgroup gauges —
to this hive's own collector, exactly like an agent does and with
no separate toggle. These come from the host, not the in-container Claude SDK,
@ -342,7 +341,7 @@ a crash and `tracing::warn!`s `"container crash detected"` — nothing else:
no broker event, no per-hive dashboard banner. Both of those used to exist;
this change removed them in favour of the log line.
When `otel.enable` is on, the hive tier's own collector runs a `journald`
The hive tier's own collector runs a `journald`
receiver scoped to `hive-c0re.service` (no bind mount needed — unlike an
agent's journal, this collector already shares a filesystem with the unit it
reads, being a host systemd service rather than a container) and forwards
@ -353,18 +352,13 @@ matches it directly and pages the operator — no agent, and no per-hive
dashboard, in the loop.
Requires `services.journald.storage = "persistent"` (the NixOS default) on
every hive host with `otel.enable` set — asserted at eval time.
⚠️ **A hive with `otel.enable` off has no operator-visible crash signal at
all** — the per-hive dashboard banner this replaces no longer exists, and
there is no fallback path. Turning on `otel.enable` is what makes a crash
reach anyone.
every hive host — asserted at eval time.
## Agent-emitted per-turn metrics (`hive-agent`)
<!-- vale write-good.Passive = NO -->
When OTEL is enabled, the harness itself (`hive-agent`) exports one small set
The harness itself (`hive-agent`) exports one small set
of metrics per claude turn, recorded the moment the turn ends (not polled).
These are deliberately the fields Claude Code's own built-in export (see
above) can't know about — the harness's own wall-clock timing, what woke the
@ -508,8 +502,7 @@ and the start of the response body. It repeats a persisting failure every
## Agent-emitted custom metrics (`hive-metric`)
Agents can push arbitrary labeled metrics to the same OTEL collector via the
`hive-metric` CLI tool, available in every agent container when
`services.hyperhive.otel.enable = true`.
`hive-metric` CLI tool, available in every agent container.
### Usage
@ -544,11 +537,11 @@ hive-metric queue_depth 17 --type gauge
hive-metric api_latency_ms 142.5 --type gauge --labels model=sonnet --labels tier=api
```
### Error when OTEL isn't configured
### Error when an agent turns OTEL off
When `services.hyperhive.otel.enable = false` (the default), the
`OTEL_EXPORTER_OTLP_ENDPOINT` env var isn't set and `hive-metric` exits
with an informative error message. No silently dropped metrics.
An agent whose own config sets `services.hyperhive.agent.otel.enable = false`
has no `OTEL_EXPORTER_OTLP_ENDPOINT` env var, and `hive-metric` exits with an
informative error message. No silently dropped metrics.
### Wire format