docs/observability.md

@@ -16,41 +16,12 @@ services.hyperhive.otel = {
 };
 ```
 
-`enable` is the single gate. `endpoint` (required when enabled) is where
+`enable` is the single gate. `endpoint` (required when enabled) is the OTLP
-telemetry ultimately goes.
+HTTP endpoint; hive-c0re injects it as `OTEL_EXPORTER_OTLP_ENDPOINT` into
+every agent's systemd service via the generated meta flake.
 
-**There is exactly one way telemetry leaves a hive: through the collector that
+Each agent's harness (hive-ag3nt) exports directly to the collector — the
-`enable` starts on the host.** Agents never talk to `endpoint` themselves —
+pipeline keeps working even when hive-c0re is down.
-they export unauthenticated to a bridge address only their own containers can
-reach, and the collector forwards upstream with the auth header. So the
-upstream credential exists in one place, on the host, and no agent ever holds
-a copy.
-
-⚠️ **The collector is therefore in the path of all telemetry.** It runs on the
-same host as the agents and restarts on failure, and telemetry is not the
-control plane — degraded telemetry is not degraded operation — but the export
-no longer survives independently of anything host-side.
-
-### what the agent→collector hop is and isn't
-
-**It has no application-level auth.** The receiver takes any OTLP that reaches
-it; what bounds who can reach it is the firewall — `exposeHostPorts` opens the
-port on the bridge interface only. So "unauthenticated to a bridge address"
-means *reachable from an agent container*, not *presents a credential*.
-
-The consequence, stated because it is a choice rather than an oversight: **any
-agent can push arbitrary OTLP, and the collector forwards it upstream under the
-operator's credential.** It cannot tell a container's genuine Claude Code stats
-from anything else shaped like OTLP arriving on that port — including data
-smuggled out in resource attributes on an otherwise-legitimate export.
-
-That is a **different risk from the one the collector fixes**, and strictly
-smaller than what preceded it: before, every agent held the upstream credential
-itself, so it could do all of the above *and* use the token anywhere else. The
-collector removes the token and keeps the pipe. Agents are inside the trust
-boundary (`docs/security.md`: capability = accepted risk), so an agent being
-able to *send* is an accepted extension of that boundary — but it is not
-closed by this design, and nothing here should be read as closing it.
 
 ## Options reference
 
@@ -72,17 +43,17 @@ OTLP wire protocol, passed as `OTEL_EXPORTER_OTLP_PROTOCOL`. Accepted values:
 
 ### `services.hyperhive.otel.headersCredential` — string or null, default `null`
 
-Absolute path to a secret file on the host holding the upstream auth header as
+Absolute path to a secret file on the host whose contents become
-`NAME=value` (e.g. `Authorization=Bearer <token>`).
+`OTEL_EXPORTER_OTLP_HEADERS` (e.g. `Authorization=Bearer <token>`).
 
-**Only the host collector reads it.** It arrives as an `EnvironmentFile` on the
+hive-c0re forwards this host file into each agent container via
-collector's unit, so the value is never read by nix, never copied into the
+`systemd-nspawn --load-credential=otel-headers:<path>`; the inner harness unit
-store or the generated config, never passed in argv — and **never forwarded
+inherits it by name. The token is never copied into the nix store, generated
-into an agent container**. An agent cannot read the hive's upstream credential
+config, a bind mount, or argv.
-because it is never given one.
 
-Leave `null` if the upstream needs no auth header; the collector then sends
+Leave `null` if the endpoint needs no auth header. A configured-but-missing
-none rather than an empty one.
+file is skipped with a log warning — OTEL still exports, just without the auth
+header.
 
 ```nix
 services.hyperhive.otel = {
@@ -122,50 +93,6 @@ metrics on process exit, so interval tuning is not required for metrics to be
 exported. A lower value gives more frequent intermediate flushes within
 long-running turns — cosmetic, not a correctness knob.
 
-## The host collector
-
-`enable` starts an OpenTelemetry collector on the host. It is not optional and
-there is no second path — that is the whole point:
-
-```nix
-services.hyperhive.otel = {
-  enable            = true;
-  endpoint          = "https://collector.example.com/otel";  # the upstream
-  headersCredential = "/run/secrets/otel-headers";           # only the host reads it
-};
-```
-
-**Why it isn't a knob.** Exporting straight to `endpoint` means every agent
-needs the credential to authenticate — and the harness delivers that token into
-the agent's own `~/.claude/settings.json`, a file the agent can read. `0600`
-protects it from other containers, not from the agent itself. As long as the
-direct path stays *selectable*, that hole stays selectable; an option that can
-reintroduce it is a hole with extra steps.
-
-**`endpoint` keeps meaning "where telemetry ultimately goes."** The collector
-does not redefine it — the agent-facing value is *derived*
-(`http://<bridgeIp>:<collector.port>`), so an existing deployment's `endpoint`
-keeps working unchanged. The bridge port is contributed to `exposeHostPorts`
-automatically; there is nothing to open by hand.
-
-### `services.hyperhive.otel.collector.port` — port, default `4318`
-
-The OTLP/HTTP port the collector listens on, bound to the bridge IP only.
-
-### `services.hyperhive.otel.collector.upstreamHeaderName` — string, default `"Authorization"`
-
-Name of the header the collector sends upstream. The **value** comes from the
-credential file at runtime (`EnvironmentFile` → `${env:<name>}`), never from
-nix — so header names are config and header values are secrets, which is the
-only split the collector's static header map can express.
-
-⚠️ **`endpoint` must be valid for `protocol`.** The upstream exporter follows
-`otel.protocol` (`grpc` → the gRPC exporter, otherwise OTLP/HTTP), and the gRPC
-exporter takes an *address*: `https://host/path` is a legal
-`OTEL_EXPORTER_OTLP_ENDPOINT` for HTTP but fails as gRPC with *"missing port in
-address"*. The collector's config is validated at build time, so a mismatch is
-a build error naming the reason rather than telemetry silently going nowhere.
-
 ## Network access
 
 Agent containers can only reach the host on ports 80 and 443 by default. If
@@ -185,10 +112,6 @@ services.hyperhive.otel.endpoint = "http://10.42.0.1:4318";
 The bridge IP is the host's address on the `hvbr0` bridge, typically
 `10.42.0.1`. See `docs/network.md::Reaching host services` for details.
 
-⚠️ **You do not need either line for hyperhive's own telemetry** — `otel.enable`
-contributes the collector's port and derives the endpoint itself. The above is
-for pointing something *else* at a host-local service.
-
 ## Built-in resource labels
 
 Every agent's export includes these resource attributes automatically:

nix/host-modules/hive-c0re/environment.nix

@@ -83,29 +83,15 @@ in
     otel = config.services.hyperhive.otel;
   in
   {
-    # `otel.endpoint` means "where telemetry ultimately goes" and keeps
+    HYPERHIVE_OTEL_ENDPOINT = otel.endpoint;
-    # that meaning; what agents are handed is the *first hop*, which is
+    HYPERHIVE_OTEL_PROTOCOL = otel.protocol;
-    # always this hive's own collector. Deriving it rather than
-    # redefining `endpoint` is what lets every existing deployment keep
-    # its configured value untouched.
-    HYPERHIVE_OTEL_ENDPOINT = "http://${config.services.hyperhive.network.bridgeIp}:${toString otel.collector.port}";
-    # The first hop is the collector's OTLP/HTTP receiver, which speaks
-    # protobuf regardless of what the upstream wants — `otel.protocol`
-    # describes the *upstream* link, and the collector's own exporter is
-    # what has to honour it (see nix/host-modules/otel.nix).
-    HYPERHIVE_OTEL_PROTOCOL = "http/protobuf";
   }
   // lib.optionalAttrs (otel.extraResourceAttributes != "") {
     HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES = otel.extraResourceAttributes;
   }
-  # HYPERHIVE_OTEL_HEADERS_CREDENTIAL is deliberately NOT emitted, and
+  // lib.optionalAttrs (otel.headersCredential != null) {
-  # its absence is the security half of this design. It is the variable
+    HYPERHIVE_OTEL_HEADERS_CREDENTIAL = otel.headersCredential;
-  # that put the upstream token in an agent's own settings.json:
+  }
-  # host_config.rs forwards it into the container as an nspawn
-  # credential, and claude-settings.nix's `hive-otel-header` oneshot
-  # then writes the value into a file the agent can read. The collector
-  # holding the credential achieves nothing while the harness keeps
-  # handing out a copy — so there is exactly one holder, on the host.
   // lib.optionalAttrs (otel.metricIntervalMs != null) {
     HYPERHIVE_OTEL_METRIC_INTERVAL_MS = toString otel.metricIntervalMs;
   }

nix/host-modules/otel.nix

@@ -16,19 +16,9 @@
     enable = lib.mkEnableOption ''
       hive-wide export of every agent's Claude Code stats (token usage,
       cost, tool calls) to an OTLP endpoint via Claude Code's built-in
-      OpenTelemetry. One switch for all agents.
+      OpenTelemetry. One switch for all agents; each harness exports
-
+      directly to the collector, so it keeps working even when hive-c0re
-      Enabling this also runs a collector on the host: there is exactly
+      is down
-      one way telemetry leaves this hive, and it is through that
-      collector. Agents export unauthenticated to a bridge address only
-      their own containers can reach, and the collector is the only
-      holder of the upstream credential — an agent never sees it.
-
-      ⚠️ The collector is therefore in the path of all telemetry. It runs
-      on the same host as the agents and restarts on failure, and
-      telemetry is not the control plane, so degraded telemetry is not
-      degraded operation — but the export no longer survives independently
-      of anything host-side
     '';
 
     endpoint = lib.mkOption {
@@ -62,19 +52,16 @@
       default = null;
       example = "/run/secrets/otel-headers";
       description = ''
-        Absolute path to an operator-provided secret file holding the
+        Absolute path to an operator-provided secret file whose contents
-        upstream auth header as `NAME=value` (e.g.
+        become `OTEL_EXPORTER_OTLP_HEADERS` (e.g.
-        `Authorization=Bearer <token>`).
+        `Authorization=Bearer <token>`). hive-c0re forwards this host
-
+        file into each agent container's credential store via
-        **Only the host-side collector reads this.** It reaches the
+        systemd-nspawn `--load-credential=otel-headers:<path>`; the inner
-        collector as an `EnvironmentFile`, so the value is never read by
+        harness unit inherits it by name (`LoadCredential`), so the token
-        nix, never copied into the store or the generated config, and
+        is never copied into the nix store, the generated config, a bind
-        never passed in argv — and it is never forwarded into an agent
+        mount, or argv. Must be absolute. Leave null if the endpoint
-        container, which is the point of the collector existing. Must be
+        needs no auth header. A configured-but-missing file is skipped
-        absolute.
+        with a log warning (OTEL still exports, without the auth header).
-
-        Leave null if the upstream needs no auth header; the collector
-        then sends none rather than an empty one.
       '';
     };
 
@@ -101,41 +88,6 @@
       '';
     };
 
-    collector.upstreamHeaderName = lib.mkOption {
-      type = lib.types.str;
-      default = "Authorization";
-      description = ''
-        Name of the HTTP header the collector sends upstream, whose
-        *value* comes from `headersCredential`.
-
-        The name is here and the value is not, and that split is forced
-        rather than chosen: the collector models exporter headers as a
-        static map, so rendering them means nix reading the value — the
-        one thing `headersCredential` being a path exists to prevent.
-        A name is public, a value is not.
-
-        ⇒ exactly one header is expressible this way. A credential
-        carrying several (`a=1,b=2`) would be read as a single value,
-        which is why the shape is a named header rather than an opaque
-        blob: a second header has to be *declared*, not smuggled.
-      '';
-    };
-
-    collector.port = lib.mkOption {
-      type = lib.types.port;
-      default = 4318;
-      description = ''
-        Port the collector's OTLP/HTTP receiver listens on, at
-        `services.hyperhive.network.bridgeIp`. 4318 is the OTLP/HTTP
-        default.
-
-        The port is contributed to
-        `services.hyperhive.network.exposeHostPorts`, which opens it on
-        the bridge interface only — so it is reachable from agent
-        containers and not from the outside world.
-      '';
-    };
-
     metricIntervalMs = lib.mkOption {
       type = lib.types.nullOr lib.types.ints.positive;
       default = null;
@@ -154,79 +106,12 @@
     };
   };
 
-  config = lib.mkMerge [
+  config = lib.mkIf config.services.hyperhive.c0re.enable {
-    (lib.mkIf config.services.hyperhive.c0re.enable {
+    assertions = lib.optionals config.services.hyperhive.otel.enable [
-      assertions = lib.optionals config.services.hyperhive.otel.enable [
+      {
-        {
+        assertion = config.services.hyperhive.otel.endpoint != "";
-          assertion = config.services.hyperhive.otel.endpoint != "";
+        message = "services.hyperhive.otel.enable is true but services.hyperhive.otel.endpoint is empty.";
-          message = "services.hyperhive.otel.enable is true but services.hyperhive.otel.endpoint is empty.";
+      }
-        }
+    ];
-      ];
+  };
-    })
-
-    (lib.mkIf config.services.hyperhive.otel.enable (
-      (
-        let
-          otel = config.services.hyperhive.otel;
-          listen = "${config.services.hyperhive.network.bridgeIp}:${toString otel.collector.port}";
-          # `otel.protocol` describes the UPSTREAM link and always did.
-          # Inserting a collector splits the path in two, and the
-          # upstream half is the one that has to keep honouring it — so
-          # the exporter is chosen by it, rather than the option quietly
-          # becoming "how agents talk to the collector". The agent half
-          # is pinned to OTLP/HTTP by the receiver below (derived in
-          # hive-c0re/environment.nix).
-          grpcUpstream = otel.protocol == "grpc";
-          upstreamName = if grpcUpstream then "otlp" else "otlphttp";
-          upstream = {
-            endpoint = otel.endpoint;
-          }
-          // lib.optionalAttrs (otel.headersCredential != null) {
-            # The value is interpolated by the collector at runtime from
-            # its environment, never by nix. `EnvironmentFile` below is
-            # what puts it there. No credential configured means no
-            # header at all — an upstream that needs no auth is a
-            # legitimate deployment, and rendering `${env:…}` for a
-            # variable nothing sets would send the literal.
-            headers.${otel.collector.upstreamHeaderName} = "\${env:${otel.collector.upstreamHeaderName}}";
-          }
-          // lib.optionalAttrs (otel.protocol == "http/json") { encoding = "json"; };
-        in
-        {
-          # Reachable from agent containers and nowhere else: this opens
-          # the port on the bridge interface only.
-          services.hyperhive.network.exposeHostPorts = [ otel.collector.port ];
-
-          services.opentelemetry-collector = {
-            enable = true;
-            # `validateConfigFile` defaults to `isStorePath configFile`,
-            # and `configFile` is null on the `settings` path — so the
-            # upstream default is OFF for exactly the way this module
-            # configures it. Turning it on runs `otelcol validate` at
-            # build time, which is the collector checking its own config.
-            # ⚠️ It parses; it does not prove a sample arrives.
-            validateConfigFile = true;
-            settings = {
-              receivers.otlp.protocols.http.endpoint = listen;
-              exporters.${upstreamName} = upstream;
-              service.pipelines.metrics = {
-                receivers = [ "otlp" ];
-                exporters = [ upstreamName ];
-              };
-            };
-          };
-
-          # The credential file is already `NAME=value`, which is
-          # systemd's EnvironmentFile format — so the secret reaches the
-          # process as an environment variable without ever being read by
-          # nix, written to the store, or passed in argv.
-          systemd.services.opentelemetry-collector.serviceConfig =
-            lib.optionalAttrs (otel.headersCredential != null)
-              {
-                EnvironmentFile = otel.headersCredential;
-              };
-        }
-      )
-    ))
-  ];
 }