docs(otel): say what the agent->collector hop is and is not

argus on #3280: the receiver has no auth extension - the nixpkgs module
passes settings straight through and nothing wires one on - so
'unauthenticated to a bridge address' means reachable from an agent
container, not presents a credential. The firewall is the whole access
control.

Consequence, stated because it is a choice rather than an oversight:
any agent can push arbitrary OTLP and the collector forwards it under
the operator's credential, including data smuggled out in resource
attributes. That is a different risk from the one the collector fixes,
and strictly smaller than what preceded it - before, every agent held
the credential itself and could do all of that plus use the token
anywhere else. The collector removes the token and keeps the pipe.

Same principle this PR already applies to the availability trade: state
it where the reader is, rather than let it be discovered.

docs/observability.md

@@ -16,12 +16,41 @@ services.hyperhive.otel = {
 };
 ```
 
-`enable` is the single gate. `endpoint` (required when enabled) is the OTLP
-HTTP endpoint; hive-c0re injects it as `OTEL_EXPORTER_OTLP_ENDPOINT` into
-every agent's systemd service via the generated meta flake.
+`enable` is the single gate. `endpoint` (required when enabled) is where
+telemetry ultimately goes.
 
-Each agent's harness (hive-ag3nt) exports directly to the collector — the
-pipeline keeps working even when hive-c0re is down.
+**There is exactly one way telemetry leaves a hive: through the collector that
+`enable` starts on the host.** Agents never talk to `endpoint` themselves —
+they export unauthenticated to a bridge address only their own containers can
+reach, and the collector forwards upstream with the auth header. So the
+upstream credential exists in one place, on the host, and no agent ever holds
+a copy.
+
+⚠️ **The collector is therefore in the path of all telemetry.** It runs on the
+same host as the agents and restarts on failure, and telemetry is not the
+control plane — degraded telemetry is not degraded operation — but the export
+no longer survives independently of anything host-side.
+
+### what the agent→collector hop is and isn't
+
+**It has no application-level auth.** The receiver takes any OTLP that reaches
+it; what bounds who can reach it is the firewall — `exposeHostPorts` opens the
+port on the bridge interface only. So "unauthenticated to a bridge address"
+means *reachable from an agent container*, not *presents a credential*.
+
+The consequence, stated because it is a choice rather than an oversight: **any
+agent can push arbitrary OTLP, and the collector forwards it upstream under the
+operator's credential.** It cannot tell a container's genuine Claude Code stats
+from anything else shaped like OTLP arriving on that port — including data
+smuggled out in resource attributes on an otherwise-legitimate export.
+
+That is a **different risk from the one the collector fixes**, and strictly
+smaller than what preceded it: before, every agent held the upstream credential
+itself, so it could do all of the above *and* use the token anywhere else. The
+collector removes the token and keeps the pipe. Agents are inside the trust
+boundary (`docs/security.md`: capability = accepted risk), so an agent being
+able to *send* is an accepted extension of that boundary — but it is not
+closed by this design, and nothing here should be read as closing it.
 
 ## Options reference
 
@@ -43,17 +72,17 @@ OTLP wire protocol, passed as `OTEL_EXPORTER_OTLP_PROTOCOL`. Accepted values:
 
 ### `services.hyperhive.otel.headersCredential` — string or null, default `null`
 
-Absolute path to a secret file on the host whose contents become
-`OTEL_EXPORTER_OTLP_HEADERS` (e.g. `Authorization=Bearer <token>`).
+Absolute path to a secret file on the host holding the upstream auth header as
+`NAME=value` (e.g. `Authorization=Bearer <token>`).
 
-hive-c0re forwards this host file into each agent container via
-`systemd-nspawn --load-credential=otel-headers:<path>`; the inner harness unit
-inherits it by name. The token is never copied into the nix store, generated
-config, a bind mount, or argv.
+**Only the host collector reads it.** It arrives as an `EnvironmentFile` on the
+collector's unit, so the value is never read by nix, never copied into the
+store or the generated config, never passed in argv — and **never forwarded
+into an agent container**. An agent cannot read the hive's upstream credential
+because it is never given one.
 
-Leave `null` if the endpoint needs no auth header. A configured-but-missing
-file is skipped with a log warning — OTEL still exports, just without the auth
-header.
+Leave `null` if the upstream needs no auth header; the collector then sends
+none rather than an empty one.
 
 ```nix
 services.hyperhive.otel = {
@@ -93,6 +122,50 @@ metrics on process exit, so interval tuning is not required for metrics to be
 exported. A lower value gives more frequent intermediate flushes within
 long-running turns — cosmetic, not a correctness knob.
 
+## The host collector
+
+`enable` starts an OpenTelemetry collector on the host. It is not optional and
+there is no second path — that is the whole point:
+
+```nix
+services.hyperhive.otel = {
+  enable            = true;
+  endpoint          = "https://collector.example.com/otel";  # the upstream
+  headersCredential = "/run/secrets/otel-headers";           # only the host reads it
+};
+```
+
+**Why it isn't a knob.** Exporting straight to `endpoint` means every agent
+needs the credential to authenticate — and the harness delivers that token into
+the agent's own `~/.claude/settings.json`, a file the agent can read. `0600`
+protects it from other containers, not from the agent itself. As long as the
+direct path stays *selectable*, that hole stays selectable; an option that can
+reintroduce it is a hole with extra steps.
+
+**`endpoint` keeps meaning "where telemetry ultimately goes."** The collector
+does not redefine it — the agent-facing value is *derived*
+(`http://<bridgeIp>:<collector.port>`), so an existing deployment's `endpoint`
+keeps working unchanged. The bridge port is contributed to `exposeHostPorts`
+automatically; there is nothing to open by hand.
+
+### `services.hyperhive.otel.collector.port` — port, default `4318`
+
+The OTLP/HTTP port the collector listens on, bound to the bridge IP only.
+
+### `services.hyperhive.otel.collector.upstreamHeaderName` — string, default `"Authorization"`
+
+Name of the header the collector sends upstream. The **value** comes from the
+credential file at runtime (`EnvironmentFile` → `${env:<name>}`), never from
+nix — so header names are config and header values are secrets, which is the
+only split the collector's static header map can express.
+
+⚠️ **`endpoint` must be valid for `protocol`.** The upstream exporter follows
+`otel.protocol` (`grpc` → the gRPC exporter, otherwise OTLP/HTTP), and the gRPC
+exporter takes an *address*: `https://host/path` is a legal
+`OTEL_EXPORTER_OTLP_ENDPOINT` for HTTP but fails as gRPC with *"missing port in
+address"*. The collector's config is validated at build time, so a mismatch is
+a build error naming the reason rather than telemetry silently going nowhere.
+
 ## Network access
 
 Agent containers can only reach the host on ports 80 and 443 by default. If
@@ -112,6 +185,10 @@ services.hyperhive.otel.endpoint = "http://10.42.0.1:4318";
 The bridge IP is the host's address on the `hvbr0` bridge, typically
 `10.42.0.1`. See `docs/network.md::Reaching host services` for details.
 
+⚠️ **You do not need either line for hyperhive's own telemetry** — `otel.enable`
+contributes the collector's port and derives the endpoint itself. The above is
+for pointing something *else* at a host-local service.
+
 ## Built-in resource labels
 
 Every agent's export includes these resource attributes automatically:

nix/host-modules/hive-c0re/environment.nix

@@ -83,15 +83,29 @@ in
     otel = config.services.hyperhive.otel;
   in
   {
-    HYPERHIVE_OTEL_ENDPOINT = otel.endpoint;
-    HYPERHIVE_OTEL_PROTOCOL = otel.protocol;
+    # `otel.endpoint` means "where telemetry ultimately goes" and keeps
+    # that meaning; what agents are handed is the *first hop*, which is
+    # always this hive's own collector. Deriving it rather than
+    # redefining `endpoint` is what lets every existing deployment keep
+    # its configured value untouched.
+    HYPERHIVE_OTEL_ENDPOINT = "http://${config.services.hyperhive.network.bridgeIp}:${toString otel.collector.port}";
+    # The first hop is the collector's OTLP/HTTP receiver, which speaks
+    # protobuf regardless of what the upstream wants — `otel.protocol`
+    # describes the *upstream* link, and the collector's own exporter is
+    # what has to honour it (see nix/host-modules/otel.nix).
+    HYPERHIVE_OTEL_PROTOCOL = "http/protobuf";
   }
   // lib.optionalAttrs (otel.extraResourceAttributes != "") {
     HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES = otel.extraResourceAttributes;
   }
-  // lib.optionalAttrs (otel.headersCredential != null) {
-    HYPERHIVE_OTEL_HEADERS_CREDENTIAL = otel.headersCredential;
-  }
+  # HYPERHIVE_OTEL_HEADERS_CREDENTIAL is deliberately NOT emitted, and
+  # its absence is the security half of this design. It is the variable
+  # that put the upstream token in an agent's own settings.json:
+  # host_config.rs forwards it into the container as an nspawn
+  # credential, and claude-settings.nix's `hive-otel-header` oneshot
+  # then writes the value into a file the agent can read. The collector
+  # holding the credential achieves nothing while the harness keeps
+  # handing out a copy — so there is exactly one holder, on the host.
   // lib.optionalAttrs (otel.metricIntervalMs != null) {
     HYPERHIVE_OTEL_METRIC_INTERVAL_MS = toString otel.metricIntervalMs;
   }

nix/host-modules/otel.nix

@@ -16,9 +16,19 @@
     enable = lib.mkEnableOption ''
       hive-wide export of every agent's Claude Code stats (token usage,
       cost, tool calls) to an OTLP endpoint via Claude Code's built-in
-      OpenTelemetry. One switch for all agents; each harness exports
-      directly to the collector, so it keeps working even when hive-c0re
-      is down
+      OpenTelemetry. One switch for all agents.
+
+      Enabling this also runs a collector on the host: there is exactly
+      one way telemetry leaves this hive, and it is through that
+      collector. Agents export unauthenticated to a bridge address only
+      their own containers can reach, and the collector is the only
+      holder of the upstream credential — an agent never sees it.
+
+      ⚠️ The collector is therefore in the path of all telemetry. It runs
+      on the same host as the agents and restarts on failure, and
+      telemetry is not the control plane, so degraded telemetry is not
+      degraded operation — but the export no longer survives independently
+      of anything host-side
     '';
 
     endpoint = lib.mkOption {
@@ -52,16 +62,19 @@
       default = null;
       example = "/run/secrets/otel-headers";
       description = ''
-        Absolute path to an operator-provided secret file whose contents
-        become `OTEL_EXPORTER_OTLP_HEADERS` (e.g.
-        `Authorization=Bearer <token>`). hive-c0re forwards this host
-        file into each agent container's credential store via
-        systemd-nspawn `--load-credential=otel-headers:<path>`; the inner
-        harness unit inherits it by name (`LoadCredential`), so the token
-        is never copied into the nix store, the generated config, a bind
-        mount, or argv. Must be absolute. Leave null if the endpoint
-        needs no auth header. A configured-but-missing file is skipped
-        with a log warning (OTEL still exports, without the auth header).
+        Absolute path to an operator-provided secret file holding the
+        upstream auth header as `NAME=value` (e.g.
+        `Authorization=Bearer <token>`).
+
+        **Only the host-side collector reads this.** It reaches the
+        collector as an `EnvironmentFile`, so the value is never read by
+        nix, never copied into the store or the generated config, and
+        never passed in argv — and it is never forwarded into an agent
+        container, which is the point of the collector existing. Must be
+        absolute.
+
+        Leave null if the upstream needs no auth header; the collector
+        then sends none rather than an empty one.
       '';
     };
 
@@ -88,6 +101,41 @@
       '';
     };
 
+    collector.upstreamHeaderName = lib.mkOption {
+      type = lib.types.str;
+      default = "Authorization";
+      description = ''
+        Name of the HTTP header the collector sends upstream, whose
+        *value* comes from `headersCredential`.
+
+        The name is here and the value is not, and that split is forced
+        rather than chosen: the collector models exporter headers as a
+        static map, so rendering them means nix reading the value — the
+        one thing `headersCredential` being a path exists to prevent.
+        A name is public, a value is not.
+
+        ⇒ exactly one header is expressible this way. A credential
+        carrying several (`a=1,b=2`) would be read as a single value,
+        which is why the shape is a named header rather than an opaque
+        blob: a second header has to be *declared*, not smuggled.
+      '';
+    };
+
+    collector.port = lib.mkOption {
+      type = lib.types.port;
+      default = 4318;
+      description = ''
+        Port the collector's OTLP/HTTP receiver listens on, at
+        `services.hyperhive.network.bridgeIp`. 4318 is the OTLP/HTTP
+        default.
+
+        The port is contributed to
+        `services.hyperhive.network.exposeHostPorts`, which opens it on
+        the bridge interface only — so it is reachable from agent
+        containers and not from the outside world.
+      '';
+    };
+
     metricIntervalMs = lib.mkOption {
       type = lib.types.nullOr lib.types.ints.positive;
       default = null;
@@ -106,12 +154,79 @@
     };
   };
 
-  config = lib.mkIf config.services.hyperhive.c0re.enable {
-    assertions = lib.optionals config.services.hyperhive.otel.enable [
-      {
-        assertion = config.services.hyperhive.otel.endpoint != "";
-        message = "services.hyperhive.otel.enable is true but services.hyperhive.otel.endpoint is empty.";
-      }
-    ];
-  };
+  config = lib.mkMerge [
+    (lib.mkIf config.services.hyperhive.c0re.enable {
+      assertions = lib.optionals config.services.hyperhive.otel.enable [
+        {
+          assertion = config.services.hyperhive.otel.endpoint != "";
+          message = "services.hyperhive.otel.enable is true but services.hyperhive.otel.endpoint is empty.";
+        }
+      ];
+    })
+
+    (lib.mkIf config.services.hyperhive.otel.enable (
+      (
+        let
+          otel = config.services.hyperhive.otel;
+          listen = "${config.services.hyperhive.network.bridgeIp}:${toString otel.collector.port}";
+          # `otel.protocol` describes the UPSTREAM link and always did.
+          # Inserting a collector splits the path in two, and the
+          # upstream half is the one that has to keep honouring it — so
+          # the exporter is chosen by it, rather than the option quietly
+          # becoming "how agents talk to the collector". The agent half
+          # is pinned to OTLP/HTTP by the receiver below (derived in
+          # hive-c0re/environment.nix).
+          grpcUpstream = otel.protocol == "grpc";
+          upstreamName = if grpcUpstream then "otlp" else "otlphttp";
+          upstream = {
+            endpoint = otel.endpoint;
+          }
+          // lib.optionalAttrs (otel.headersCredential != null) {
+            # The value is interpolated by the collector at runtime from
+            # its environment, never by nix. `EnvironmentFile` below is
+            # what puts it there. No credential configured means no
+            # header at all — an upstream that needs no auth is a
+            # legitimate deployment, and rendering `${env:…}` for a
+            # variable nothing sets would send the literal.
+            headers.${otel.collector.upstreamHeaderName} = "\${env:${otel.collector.upstreamHeaderName}}";
+          }
+          // lib.optionalAttrs (otel.protocol == "http/json") { encoding = "json"; };
+        in
+        {
+          # Reachable from agent containers and nowhere else: this opens
+          # the port on the bridge interface only.
+          services.hyperhive.network.exposeHostPorts = [ otel.collector.port ];
+
+          services.opentelemetry-collector = {
+            enable = true;
+            # `validateConfigFile` defaults to `isStorePath configFile`,
+            # and `configFile` is null on the `settings` path — so the
+            # upstream default is OFF for exactly the way this module
+            # configures it. Turning it on runs `otelcol validate` at
+            # build time, which is the collector checking its own config.
+            # ⚠️ It parses; it does not prove a sample arrives.
+            validateConfigFile = true;
+            settings = {
+              receivers.otlp.protocols.http.endpoint = listen;
+              exporters.${upstreamName} = upstream;
+              service.pipelines.metrics = {
+                receivers = [ "otlp" ];
+                exporters = [ upstreamName ];
+              };
+            };
+          };
+
+          # The credential file is already `NAME=value`, which is
+          # systemd's EnvironmentFile format — so the secret reaches the
+          # process as an environment variable without ever being read by
+          # nix, written to the store, or passed in argv.
+          systemd.services.opentelemetry-collector.serviceConfig =
+            lib.optionalAttrs (otel.headersCredential != null)
+              {
+                EnvironmentFile = otel.headersCredential;
+              };
+        }
+      )
+    ))
+  ];
 }