diff --git a/docs/observability.md b/docs/observability.md index a8cbfbfd..13bf6864 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -16,12 +16,20 @@ services.hyperhive.otel = { }; ``` -`enable` is the single gate. `endpoint` (required when enabled) is the OTLP -HTTP endpoint; hive-c0re injects it as `OTEL_EXPORTER_OTLP_ENDPOINT` into -every agent's systemd service via the generated meta flake. +`enable` is the single gate. `endpoint` (required when enabled) is where +telemetry ultimately goes. -Each agent's harness (hive-ag3nt) exports directly to the collector — the -pipeline keeps working even when hive-c0re is down. +**There is exactly one way telemetry leaves a hive: through the collector that +`enable` starts on the host.** Agents never talk to `endpoint` themselves — +they export unauthenticated to a bridge address only their own containers can +reach, and the collector forwards upstream with the auth header. So the +upstream credential exists in one place, on the host, and no agent ever holds +a copy. + +⚠️ **The collector is therefore in the path of all telemetry.** It runs on the +same host as the agents and restarts on failure, and telemetry is not the +control plane — degraded telemetry is not degraded operation — but the export +no longer survives independently of anything host-side. ## Options reference @@ -43,17 +51,17 @@ OTLP wire protocol, passed as `OTEL_EXPORTER_OTLP_PROTOCOL`. Accepted values: ### `services.hyperhive.otel.headersCredential` — string or null, default `null` -Absolute path to a secret file on the host whose contents become -`OTEL_EXPORTER_OTLP_HEADERS` (e.g. `Authorization=Bearer `). +Absolute path to a secret file on the host holding the upstream auth header as +`NAME=value` (e.g. `Authorization=Bearer `). -hive-c0re forwards this host file into each agent container via -`systemd-nspawn --load-credential=otel-headers:`; the inner harness unit -inherits it by name. The token is never copied into the nix store, generated -config, a bind mount, or argv. +**Only the host collector reads it.** It arrives as an `EnvironmentFile` on the +collector's unit, so the value is never read by nix, never copied into the +store or the generated config, never passed in argv — and **never forwarded +into an agent container**. An agent cannot read the hive's upstream credential +because it is never given one. -Leave `null` if the endpoint needs no auth header. A configured-but-missing -file is skipped with a log warning — OTEL still exports, just without the auth -header. +Leave `null` if the upstream needs no auth header; the collector then sends +none rather than an empty one. ```nix services.hyperhive.otel = { @@ -93,44 +101,31 @@ metrics on process exit, so interval tuning is not required for metrics to be exported. A lower value gives more frequent intermediate flushes within long-running turns — cosmetic, not a correctness knob. -## Running a collector on the host (`otel.collector`) +## The host collector -By default every agent exports **straight to `endpoint`**, which means every -agent needs `headersCredential` to authenticate — and the harness delivers that -token into the agent's own `~/.claude/settings.json`, a file the agent can read. -`0600` protects it from other containers, not from the agent itself. - -With `services.hyperhive.otel.collector.enable = true`, a collector runs on the -host and holds the credential instead. Agents export **unauthenticated** to a -bridge address only their own containers can reach; the collector adds the -upstream header and forwards to `endpoint`. +`enable` starts an OpenTelemetry collector on the host. It is not optional and +there is no second path — that is the whole point: ```nix services.hyperhive.otel = { enable = true; - endpoint = "https://collector.example.com/otel"; # still the upstream - headersCredential = "/run/secrets/otel-headers"; # now only the host reads it - collector.enable = true; + endpoint = "https://collector.example.com/otel"; # the upstream + headersCredential = "/run/secrets/otel-headers"; # only the host reads it }; ``` -**`endpoint` keeps meaning "where telemetry ultimately goes."** Turning the -collector on does not redefine it — the agent-facing value is *derived* +**Why it isn't a knob.** Exporting straight to `endpoint` means every agent +needs the credential to authenticate — and the harness delivers that token into +the agent's own `~/.claude/settings.json`, a file the agent can read. `0600` +protects it from other containers, not from the agent itself. As long as the +direct path stays *selectable*, that hole stays selectable; an option that can +reintroduce it is a hole with extra steps. + +**`endpoint` keeps meaning "where telemetry ultimately goes."** The collector +does not redefine it — the agent-facing value is *derived* (`http://:`), so an existing deployment's `endpoint` -keeps working unchanged. The bridge port is contributed to -`exposeHostPorts` automatically; there is nothing to open by hand. - -⚠️ **The trade:** without a collector each harness exports directly, so -telemetry survives anything host-side being down. A local collector is a new -dependency in that path. It runs on the same host as the agents, so the window -is small — but it is not zero. - -### `services.hyperhive.otel.collector.enable` — bool, default `false` - -Off means *absent*: no unit, no port, and `endpoint` keeps its current meaning -for every agent. Requires `headersCredential` to be set — a collector with no -credential is pure indirection, and an assertion says so rather than letting it -deploy. +keeps working unchanged. The bridge port is contributed to `exposeHostPorts` +automatically; there is nothing to open by hand. ### `services.hyperhive.otel.collector.port` — port, default `4318` @@ -169,9 +164,9 @@ services.hyperhive.otel.endpoint = "http://10.42.0.1:4318"; The bridge IP is the host's address on the `hvbr0` bridge, typically `10.42.0.1`. See `docs/network.md::Reaching host services` for details. -This is the manual form of what `otel.collector.enable` does for you — with the -collector on, the port is contributed and the endpoint derived, so neither line -above is needed. +⚠️ **You do not need either line for hyperhive's own telemetry** — `otel.enable` +contributes the collector's port and derives the endpoint itself. The above is +for pointing something *else* at a host-local service. ## Built-in resource labels diff --git a/nix/host-modules/otel.nix b/nix/host-modules/otel.nix index 5a516dc8..87a07713 100644 --- a/nix/host-modules/otel.nix +++ b/nix/host-modules/otel.nix @@ -62,16 +62,19 @@ default = null; example = "/run/secrets/otel-headers"; description = '' - Absolute path to an operator-provided secret file whose contents - become `OTEL_EXPORTER_OTLP_HEADERS` (e.g. - `Authorization=Bearer `). hive-c0re forwards this host - file into each agent container's credential store via - systemd-nspawn `--load-credential=otel-headers:`; the inner - harness unit inherits it by name (`LoadCredential`), so the token - is never copied into the nix store, the generated config, a bind - mount, or argv. Must be absolute. Leave null if the endpoint - needs no auth header. A configured-but-missing file is skipped - with a log warning (OTEL still exports, without the auth header). + Absolute path to an operator-provided secret file holding the + upstream auth header as `NAME=value` (e.g. + `Authorization=Bearer `). + + **Only the host-side collector reads this.** It reaches the + collector as an `EnvironmentFile`, so the value is never read by + nix, never copied into the store or the generated config, and + never passed in argv — and it is never forwarded into an agent + container, which is the point of the collector existing. Must be + absolute. + + Leave null if the upstream needs no auth header; the collector + then sends none rather than an empty one. ''; };