docs(otel): document the collector in the page the index points at

argus's non-blocking note on #3278: docs/observability.md is what this
repo's reading-paths index names as 'what OTEL options are available',
and it did not mention collector.enable/port/upstreamHeaderName at all.
The nix docstrings covered it, but not where a reader following the
established path would look.

Carries the two things a docstring is a poor home for: that endpoint
keeps meaning 'where telemetry ultimately goes' (the agent-facing value
is derived, so an existing deployment is unaffected), and the
availability trade the collector makes against the direct-export
property this page already promises.
This commit is contained in:
atlas 2026-08-15 09:57:41 +02:00 committed by mara
commit f7fbad7655

View file

@ -93,6 +93,63 @@ metrics on process exit, so interval tuning is not required for metrics to be
exported. A lower value gives more frequent intermediate flushes within exported. A lower value gives more frequent intermediate flushes within
long-running turns — cosmetic, not a correctness knob. long-running turns — cosmetic, not a correctness knob.
## Running a collector on the host (`otel.collector`)
By default every agent exports **straight to `endpoint`**, which means every
agent needs `headersCredential` to authenticate — and the harness delivers that
token into the agent's own `~/.claude/settings.json`, a file the agent can read.
`0600` protects it from other containers, not from the agent itself.
With `services.hyperhive.otel.collector.enable = true`, a collector runs on the
host and holds the credential instead. Agents export **unauthenticated** to a
bridge address only their own containers can reach; the collector adds the
upstream header and forwards to `endpoint`.
```nix
services.hyperhive.otel = {
enable = true;
endpoint = "https://collector.example.com/otel"; # still the upstream
headersCredential = "/run/secrets/otel-headers"; # now only the host reads it
collector.enable = true;
};
```
**`endpoint` keeps meaning "where telemetry ultimately goes."** Turning the
collector on does not redefine it — the agent-facing value is *derived*
(`http://<bridgeIp>:<collector.port>`), so an existing deployment's `endpoint`
keeps working unchanged. The bridge port is contributed to
`exposeHostPorts` automatically; there is nothing to open by hand.
⚠️ **The trade:** without a collector each harness exports directly, so
telemetry survives anything host-side being down. A local collector is a new
dependency in that path. It runs on the same host as the agents, so the window
is small — but it is not zero.
### `services.hyperhive.otel.collector.enable` — bool, default `false`
Off means *absent*: no unit, no port, and `endpoint` keeps its current meaning
for every agent. Requires `headersCredential` to be set — a collector with no
credential is pure indirection, and an assertion says so rather than letting it
deploy.
### `services.hyperhive.otel.collector.port` — port, default `4318`
The OTLP/HTTP port the collector listens on, bound to the bridge IP only.
### `services.hyperhive.otel.collector.upstreamHeaderName` — string, default `"Authorization"`
Name of the header the collector sends upstream. The **value** comes from the
credential file at runtime (`EnvironmentFile``${env:<name>}`), never from
nix — so header names are config and header values are secrets, which is the
only split the collector's static header map can express.
⚠️ **`endpoint` must be valid for `protocol`.** The upstream exporter follows
`otel.protocol` (`grpc` → the gRPC exporter, otherwise OTLP/HTTP), and the gRPC
exporter takes an *address*: `https://host/path` is a legal
`OTEL_EXPORTER_OTLP_ENDPOINT` for HTTP but fails as gRPC with *"missing port in
address"*. The collector's config is validated at build time, so a mismatch is
a build error naming the reason rather than telemetry silently going nowhere.
## Network access ## Network access
Agent containers can only reach the host on ports 80 and 443 by default. If Agent containers can only reach the host on ports 80 and 443 by default. If
@ -112,6 +169,10 @@ services.hyperhive.otel.endpoint = "http://10.42.0.1:4318";
The bridge IP is the host's address on the `hvbr0` bridge, typically The bridge IP is the host's address on the `hvbr0` bridge, typically
`10.42.0.1`. See `docs/network.md::Reaching host services` for details. `10.42.0.1`. See `docs/network.md::Reaching host services` for details.
This is the manual form of what `otel.collector.enable` does for you — with the
collector on, the port is contributed and the endpoint derived, so neither line
above is needed.
## Built-in resource labels ## Built-in resource labels
Every agent's export includes these resource attributes automatically: Every agent's export includes these resource attributes automatically: