diff --git a/docs/observability.md b/docs/observability.md index 28350376..5a3fbea0 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -16,12 +16,41 @@ services.hyperhive.otel = { }; ``` -`enable` is the single gate. `endpoint` (required when enabled) is the OTLP -HTTP endpoint; hive-c0re injects it as `OTEL_EXPORTER_OTLP_ENDPOINT` into -every agent's systemd service via the generated meta flake. +`enable` is the single gate. `endpoint` (required when enabled) is where +telemetry ultimately goes. -Each agent's harness (hive-ag3nt) exports directly to the collector — the -pipeline keeps working even when hive-c0re is down. +**There is exactly one way telemetry leaves a hive: through the collector that +`enable` starts on the host.** Agents never talk to `endpoint` themselves — +they export unauthenticated to a bridge address only their own containers can +reach, and the collector forwards upstream with the auth header. So the +upstream credential exists in one place, on the host, and no agent ever holds +a copy. + +⚠️ **The collector is therefore in the path of all telemetry.** It runs on the +same host as the agents and restarts on failure, and telemetry is not the +control plane — degraded telemetry is not degraded operation — but the export +no longer survives independently of anything host-side. + +### what the agent→collector hop is and isn't + +**It has no application-level auth.** The receiver takes any OTLP that reaches +it; what bounds who can reach it is the firewall — `exposeHostPorts` opens the +port on the bridge interface only. So "unauthenticated to a bridge address" +means *reachable from an agent container*, not *presents a credential*. + +The consequence, stated because it is a choice rather than an oversight: **any +agent can push arbitrary OTLP, and the collector forwards it upstream under the +operator's credential.** It cannot tell a container's genuine Claude Code stats +from anything else shaped like OTLP arriving on that port — including data +smuggled out in resource attributes on an otherwise-legitimate export. + +That is a **different risk from the one the collector fixes**, and strictly +smaller than what preceded it: before, every agent held the upstream credential +itself, so it could do all of the above *and* use the token anywhere else. The +collector removes the token and keeps the pipe. Agents are inside the trust +boundary (`docs/security.md`: capability = accepted risk), so an agent being +able to *send* is an accepted extension of that boundary — but it is not +closed by this design, and nothing here should be read as closing it. ## Options reference @@ -43,17 +72,17 @@ OTLP wire protocol, passed as `OTEL_EXPORTER_OTLP_PROTOCOL`. Accepted values: ### `services.hyperhive.otel.headersCredential` — string or null, default `null` -Absolute path to a secret file on the host whose contents become -`OTEL_EXPORTER_OTLP_HEADERS` (e.g. `Authorization=Bearer `). +Absolute path to a secret file on the host holding the upstream auth header as +`NAME=value` (e.g. `Authorization=Bearer `). -hive-c0re forwards this host file into each agent container via -`systemd-nspawn --load-credential=otel-headers:`; the inner harness unit -inherits it by name. The token is never copied into the nix store, generated -config, a bind mount, or argv. +**Only the host collector reads it.** It arrives as an `EnvironmentFile` on the +collector's unit, so the value is never read by nix, never copied into the +store or the generated config, never passed in argv — and **never forwarded +into an agent container**. An agent cannot read the hive's upstream credential +because it is never given one. -Leave `null` if the endpoint needs no auth header. A configured-but-missing -file is skipped with a log warning — OTEL still exports, just without the auth -header. +Leave `null` if the upstream needs no auth header; the collector then sends +none rather than an empty one. ```nix services.hyperhive.otel = { @@ -93,6 +122,50 @@ metrics on process exit, so interval tuning is not required for metrics to be exported. A lower value gives more frequent intermediate flushes within long-running turns — cosmetic, not a correctness knob. +## The host collector + +`enable` starts an OpenTelemetry collector on the host. It is not optional and +there is no second path — that is the whole point: + +```nix +services.hyperhive.otel = { + enable = true; + endpoint = "https://collector.example.com/otel"; # the upstream + headersCredential = "/run/secrets/otel-headers"; # only the host reads it +}; +``` + +**Why it isn't a knob.** Exporting straight to `endpoint` means every agent +needs the credential to authenticate — and the harness delivers that token into +the agent's own `~/.claude/settings.json`, a file the agent can read. `0600` +protects it from other containers, not from the agent itself. As long as the +direct path stays *selectable*, that hole stays selectable; an option that can +reintroduce it is a hole with extra steps. + +**`endpoint` keeps meaning "where telemetry ultimately goes."** The collector +does not redefine it — the agent-facing value is *derived* +(`http://:`), so an existing deployment's `endpoint` +keeps working unchanged. The bridge port is contributed to `exposeHostPorts` +automatically; there is nothing to open by hand. + +### `services.hyperhive.otel.collector.port` — port, default `4318` + +The OTLP/HTTP port the collector listens on, bound to the bridge IP only. + +### `services.hyperhive.otel.collector.upstreamHeaderName` — string, default `"Authorization"` + +Name of the header the collector sends upstream. The **value** comes from the +credential file at runtime (`EnvironmentFile` → `${env:}`), never from +nix — so header names are config and header values are secrets, which is the +only split the collector's static header map can express. + +⚠️ **`endpoint` must be valid for `protocol`.** The upstream exporter follows +`otel.protocol` (`grpc` → the gRPC exporter, otherwise OTLP/HTTP), and the gRPC +exporter takes an *address*: `https://host/path` is a legal +`OTEL_EXPORTER_OTLP_ENDPOINT` for HTTP but fails as gRPC with *"missing port in +address"*. The collector's config is validated at build time, so a mismatch is +a build error naming the reason rather than telemetry silently going nowhere. + ## Network access Agent containers can only reach the host on ports 80 and 443 by default. If @@ -112,6 +185,10 @@ services.hyperhive.otel.endpoint = "http://10.42.0.1:4318"; The bridge IP is the host's address on the `hvbr0` bridge, typically `10.42.0.1`. See `docs/network.md::Reaching host services` for details. +⚠️ **You do not need either line for hyperhive's own telemetry** — `otel.enable` +contributes the collector's port and derives the endpoint itself. The above is +for pointing something *else* at a host-local service. + ## Built-in resource labels Every agent's export includes these resource attributes automatically: diff --git a/nix/host-modules/hive-c0re/environment.nix b/nix/host-modules/hive-c0re/environment.nix index 3046f46c..46be1d8e 100644 --- a/nix/host-modules/hive-c0re/environment.nix +++ b/nix/host-modules/hive-c0re/environment.nix @@ -83,15 +83,29 @@ in otel = config.services.hyperhive.otel; in { - HYPERHIVE_OTEL_ENDPOINT = otel.endpoint; - HYPERHIVE_OTEL_PROTOCOL = otel.protocol; + # `otel.endpoint` means "where telemetry ultimately goes" and keeps + # that meaning; what agents are handed is the *first hop*, which is + # always this hive's own collector. Deriving it rather than + # redefining `endpoint` is what lets every existing deployment keep + # its configured value untouched. + HYPERHIVE_OTEL_ENDPOINT = "http://${config.services.hyperhive.network.bridgeIp}:${toString otel.collector.port}"; + # The first hop is the collector's OTLP/HTTP receiver, which speaks + # protobuf regardless of what the upstream wants — `otel.protocol` + # describes the *upstream* link, and the collector's own exporter is + # what has to honour it (see nix/host-modules/otel.nix). + HYPERHIVE_OTEL_PROTOCOL = "http/protobuf"; } // lib.optionalAttrs (otel.extraResourceAttributes != "") { HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES = otel.extraResourceAttributes; } - // lib.optionalAttrs (otel.headersCredential != null) { - HYPERHIVE_OTEL_HEADERS_CREDENTIAL = otel.headersCredential; - } + # HYPERHIVE_OTEL_HEADERS_CREDENTIAL is deliberately NOT emitted, and + # its absence is the security half of this design. It is the variable + # that put the upstream token in an agent's own settings.json: + # host_config.rs forwards it into the container as an nspawn + # credential, and claude-settings.nix's `hive-otel-header` oneshot + # then writes the value into a file the agent can read. The collector + # holding the credential achieves nothing while the harness keeps + # handing out a copy — so there is exactly one holder, on the host. // lib.optionalAttrs (otel.metricIntervalMs != null) { HYPERHIVE_OTEL_METRIC_INTERVAL_MS = toString otel.metricIntervalMs; } diff --git a/nix/host-modules/otel.nix b/nix/host-modules/otel.nix index d12da00b..87a07713 100644 --- a/nix/host-modules/otel.nix +++ b/nix/host-modules/otel.nix @@ -16,9 +16,19 @@ enable = lib.mkEnableOption '' hive-wide export of every agent's Claude Code stats (token usage, cost, tool calls) to an OTLP endpoint via Claude Code's built-in - OpenTelemetry. One switch for all agents; each harness exports - directly to the collector, so it keeps working even when hive-c0re - is down + OpenTelemetry. One switch for all agents. + + Enabling this also runs a collector on the host: there is exactly + one way telemetry leaves this hive, and it is through that + collector. Agents export unauthenticated to a bridge address only + their own containers can reach, and the collector is the only + holder of the upstream credential — an agent never sees it. + + ⚠️ The collector is therefore in the path of all telemetry. It runs + on the same host as the agents and restarts on failure, and + telemetry is not the control plane, so degraded telemetry is not + degraded operation — but the export no longer survives independently + of anything host-side ''; endpoint = lib.mkOption { @@ -52,16 +62,19 @@ default = null; example = "/run/secrets/otel-headers"; description = '' - Absolute path to an operator-provided secret file whose contents - become `OTEL_EXPORTER_OTLP_HEADERS` (e.g. - `Authorization=Bearer `). hive-c0re forwards this host - file into each agent container's credential store via - systemd-nspawn `--load-credential=otel-headers:`; the inner - harness unit inherits it by name (`LoadCredential`), so the token - is never copied into the nix store, the generated config, a bind - mount, or argv. Must be absolute. Leave null if the endpoint - needs no auth header. A configured-but-missing file is skipped - with a log warning (OTEL still exports, without the auth header). + Absolute path to an operator-provided secret file holding the + upstream auth header as `NAME=value` (e.g. + `Authorization=Bearer `). + + **Only the host-side collector reads this.** It reaches the + collector as an `EnvironmentFile`, so the value is never read by + nix, never copied into the store or the generated config, and + never passed in argv — and it is never forwarded into an agent + container, which is the point of the collector existing. Must be + absolute. + + Leave null if the upstream needs no auth header; the collector + then sends none rather than an empty one. ''; }; @@ -88,6 +101,41 @@ ''; }; + collector.upstreamHeaderName = lib.mkOption { + type = lib.types.str; + default = "Authorization"; + description = '' + Name of the HTTP header the collector sends upstream, whose + *value* comes from `headersCredential`. + + The name is here and the value is not, and that split is forced + rather than chosen: the collector models exporter headers as a + static map, so rendering them means nix reading the value — the + one thing `headersCredential` being a path exists to prevent. + A name is public, a value is not. + + ⇒ exactly one header is expressible this way. A credential + carrying several (`a=1,b=2`) would be read as a single value, + which is why the shape is a named header rather than an opaque + blob: a second header has to be *declared*, not smuggled. + ''; + }; + + collector.port = lib.mkOption { + type = lib.types.port; + default = 4318; + description = '' + Port the collector's OTLP/HTTP receiver listens on, at + `services.hyperhive.network.bridgeIp`. 4318 is the OTLP/HTTP + default. + + The port is contributed to + `services.hyperhive.network.exposeHostPorts`, which opens it on + the bridge interface only — so it is reachable from agent + containers and not from the outside world. + ''; + }; + metricIntervalMs = lib.mkOption { type = lib.types.nullOr lib.types.ints.positive; default = null; @@ -106,12 +154,79 @@ }; }; - config = lib.mkIf config.services.hyperhive.c0re.enable { - assertions = lib.optionals config.services.hyperhive.otel.enable [ - { - assertion = config.services.hyperhive.otel.endpoint != ""; - message = "services.hyperhive.otel.enable is true but services.hyperhive.otel.endpoint is empty."; - } - ]; - }; + config = lib.mkMerge [ + (lib.mkIf config.services.hyperhive.c0re.enable { + assertions = lib.optionals config.services.hyperhive.otel.enable [ + { + assertion = config.services.hyperhive.otel.endpoint != ""; + message = "services.hyperhive.otel.enable is true but services.hyperhive.otel.endpoint is empty."; + } + ]; + }) + + (lib.mkIf config.services.hyperhive.otel.enable ( + ( + let + otel = config.services.hyperhive.otel; + listen = "${config.services.hyperhive.network.bridgeIp}:${toString otel.collector.port}"; + # `otel.protocol` describes the UPSTREAM link and always did. + # Inserting a collector splits the path in two, and the + # upstream half is the one that has to keep honouring it — so + # the exporter is chosen by it, rather than the option quietly + # becoming "how agents talk to the collector". The agent half + # is pinned to OTLP/HTTP by the receiver below (derived in + # hive-c0re/environment.nix). + grpcUpstream = otel.protocol == "grpc"; + upstreamName = if grpcUpstream then "otlp" else "otlphttp"; + upstream = { + endpoint = otel.endpoint; + } + // lib.optionalAttrs (otel.headersCredential != null) { + # The value is interpolated by the collector at runtime from + # its environment, never by nix. `EnvironmentFile` below is + # what puts it there. No credential configured means no + # header at all — an upstream that needs no auth is a + # legitimate deployment, and rendering `${env:…}` for a + # variable nothing sets would send the literal. + headers.${otel.collector.upstreamHeaderName} = "\${env:${otel.collector.upstreamHeaderName}}"; + } + // lib.optionalAttrs (otel.protocol == "http/json") { encoding = "json"; }; + in + { + # Reachable from agent containers and nowhere else: this opens + # the port on the bridge interface only. + services.hyperhive.network.exposeHostPorts = [ otel.collector.port ]; + + services.opentelemetry-collector = { + enable = true; + # `validateConfigFile` defaults to `isStorePath configFile`, + # and `configFile` is null on the `settings` path — so the + # upstream default is OFF for exactly the way this module + # configures it. Turning it on runs `otelcol validate` at + # build time, which is the collector checking its own config. + # ⚠️ It parses; it does not prove a sample arrives. + validateConfigFile = true; + settings = { + receivers.otlp.protocols.http.endpoint = listen; + exporters.${upstreamName} = upstream; + service.pipelines.metrics = { + receivers = [ "otlp" ]; + exporters = [ upstreamName ]; + }; + }; + }; + + # The credential file is already `NAME=value`, which is + # systemd's EnvironmentFile format — so the secret reaches the + # process as an environment variable without ever being read by + # nix, written to the store, or passed in argv. + systemd.services.opentelemetry-collector.serviceConfig = + lib.optionalAttrs (otel.headersCredential != null) + { + EnvironmentFile = otel.headersCredential; + }; + } + ) + )) + ]; }