diff --git a/docs/observability.md b/docs/observability.md index 28350376..a8cbfbfd 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -93,6 +93,63 @@ metrics on process exit, so interval tuning is not required for metrics to be exported. A lower value gives more frequent intermediate flushes within long-running turns — cosmetic, not a correctness knob. +## Running a collector on the host (`otel.collector`) + +By default every agent exports **straight to `endpoint`**, which means every +agent needs `headersCredential` to authenticate — and the harness delivers that +token into the agent's own `~/.claude/settings.json`, a file the agent can read. +`0600` protects it from other containers, not from the agent itself. + +With `services.hyperhive.otel.collector.enable = true`, a collector runs on the +host and holds the credential instead. Agents export **unauthenticated** to a +bridge address only their own containers can reach; the collector adds the +upstream header and forwards to `endpoint`. + +```nix +services.hyperhive.otel = { + enable = true; + endpoint = "https://collector.example.com/otel"; # still the upstream + headersCredential = "/run/secrets/otel-headers"; # now only the host reads it + collector.enable = true; +}; +``` + +**`endpoint` keeps meaning "where telemetry ultimately goes."** Turning the +collector on does not redefine it — the agent-facing value is *derived* +(`http://:`), so an existing deployment's `endpoint` +keeps working unchanged. The bridge port is contributed to +`exposeHostPorts` automatically; there is nothing to open by hand. + +⚠️ **The trade:** without a collector each harness exports directly, so +telemetry survives anything host-side being down. A local collector is a new +dependency in that path. It runs on the same host as the agents, so the window +is small — but it is not zero. + +### `services.hyperhive.otel.collector.enable` — bool, default `false` + +Off means *absent*: no unit, no port, and `endpoint` keeps its current meaning +for every agent. Requires `headersCredential` to be set — a collector with no +credential is pure indirection, and an assertion says so rather than letting it +deploy. + +### `services.hyperhive.otel.collector.port` — port, default `4318` + +The OTLP/HTTP port the collector listens on, bound to the bridge IP only. + +### `services.hyperhive.otel.collector.upstreamHeaderName` — string, default `"Authorization"` + +Name of the header the collector sends upstream. The **value** comes from the +credential file at runtime (`EnvironmentFile` → `${env:}`), never from +nix — so header names are config and header values are secrets, which is the +only split the collector's static header map can express. + +⚠️ **`endpoint` must be valid for `protocol`.** The upstream exporter follows +`otel.protocol` (`grpc` → the gRPC exporter, otherwise OTLP/HTTP), and the gRPC +exporter takes an *address*: `https://host/path` is a legal +`OTEL_EXPORTER_OTLP_ENDPOINT` for HTTP but fails as gRPC with *"missing port in +address"*. The collector's config is validated at build time, so a mismatch is +a build error naming the reason rather than telemetry silently going nowhere. + ## Network access Agent containers can only reach the host on ports 80 and 443 by default. If @@ -112,6 +169,10 @@ services.hyperhive.otel.endpoint = "http://10.42.0.1:4318"; The bridge IP is the host's address on the `hvbr0` bridge, typically `10.42.0.1`. See `docs/network.md::Reaching host services` for details. +This is the manual form of what `otel.collector.enable` does for you — with the +collector on, the port is contributed and the endpoint derived, so neither line +above is needed. + ## Built-in resource labels Every agent's export includes these resource attributes automatically: