diff --git a/docs/network.md b/docs/network.md index 1e40c4b2..5ec8cd3b 100644 --- a/docs/network.md +++ b/docs/network.md @@ -177,8 +177,11 @@ namespace. ### Reaching host services (`exposeHostPorts`) By default agents can only reach the host on 80/443 (+53 DNS), so a -host-side service on another port — e.g. a dev OTEL collector for -`services.hyperhive.otel.endpoint` (see `docs/observability.md`) — is unreachable. +host-side service on another port — e.g. a dev OTLP collector you want +agents to reach directly — is unreachable. (hyperhive's own telemetry +needs none of this: `otel.enable` opens its collector's port itself, and +`otel.endpoint` is the *upstream*, which no agent ever dials. See +`docs/observability.md`.) `services.hyperhive.network.exposeHostPorts = [ 4318 ];` opens each listed TCP port `P` on the bridge-interface `allowedTCPPorts`, so an diff --git a/docs/observability.md b/docs/observability.md index 5e4f5b01..f2ec11d4 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -16,18 +16,30 @@ services.hyperhive.otel = { }; ``` -`enable` is the single gate. `endpoint` is where telemetry goes upstream — -required when enabled *unless* this host runs the swarm's own metrics store -(`swarm.victoriametrics.enable`), which is a destination in its own right. With -both, telemetry goes to both. See +`enable` is the single gate. `endpoint` is where telemetry ends up after it +leaves the swarm — optional, because the swarm's own metrics store +(`swarm.victoriametrics.enable`) is a destination in its own right. With both, +telemetry goes to both. See [`swarm/services.md`](swarm/services.md#metrics-victoriametrics--grafana). **There is exactly one way telemetry leaves a hive: through the collector that `enable` starts on the host.** Agents never talk to `endpoint` themselves — they export unauthenticated to a bridge address only their own containers can -reach, and the collector forwards upstream with the auth header. So the -upstream credential exists in one place, on the host, and no agent ever holds -a copy. +reach. That collector forwards to the swarm's +([`swarm/services.md`](swarm/services.md#telemetry-collector-otel)), which is +the single process holding the upstream credential and the only writer to the +swarm's store. No agent holds a copy, and neither does this hive. + +⚠️ **On a hive that does not run the swarm's services, say where that swarm +collector is:** + +```nix +services.hyperhive.swarm.otel.url = "http://services-host.example:4319"; +``` + +Left unset it points at this host, where nothing is listening — the collector +starts, agents export happily, and the samples go nowhere. The service host +itself needs no such line. ⚠️ **The collector is therefore in the path of all telemetry.** It runs on the same host as the agents and restarts on failure, and telemetry is not the @@ -42,8 +54,8 @@ port on the bridge interface only. So "unauthenticated to a bridge address" means *reachable from an agent container*, not *presents a credential*. The consequence, stated because it is a choice rather than an oversight: **any -agent can push arbitrary OTLP, and the collector forwards it upstream under the -operator's credential.** It cannot tell a container's genuine Claude Code stats +agent can push arbitrary OTLP, and it is forwarded on under the operator's +credential.** Neither tier can tell a container's genuine Claude Code stats from anything else shaped like OTLP arriving on that port — including data smuggled out in resource attributes on an otherwise-legitimate export. @@ -61,33 +73,44 @@ closed by this design, and nothing here should be read as closing it. Master switch. When true, all other options below take effect. -### `services.hyperhive.otel.endpoint` — string, required when enabled unless the swarm store runs here +### `services.hyperhive.otel.endpoint` — string, default `""` -Upstream OTLP endpoint URL. Set as `OTEL_EXPORTER_OTLP_ENDPOINT` for every -agent. Example: `"https://collector.example.com/otel"`. +Upstream OTLP endpoint URL, read by the swarm's collector. Example: +`"https://collector.example.com/otel"`. -Leave it empty **only** on a host running `swarm.victoriametrics.enable` — the -local store is then the destination and the collector writes there instead. -With neither, `enable` is refused at eval: telemetry with nowhere to go is a -misconfiguration, not a quiet no-op. +Leave it empty on a swarm running its own metrics store — that store is then +the destination. With neither, the swarm collector is refused at eval: +telemetry with nowhere to go is a misconfiguration, not a quiet no-op. + +Not what agents are handed. Their endpoint is this hive's own collector, +derived from the bridge address, so setting this changes where telemetry +*ends up* and never what a container is told. ### `services.hyperhive.otel.protocol` — enum, default `"http/protobuf"` -OTLP wire protocol, passed as `OTEL_EXPORTER_OTLP_PROTOCOL`. Accepted values: +Wire protocol for the **upstream** link, honoured by the swarm collector's +exporter. Accepted values: - `"http/protobuf"` (default) - `"http/json"` - `"grpc"` +Agents are not affected: their first hop is this hive's collector, whose +OTLP/HTTP receiver takes protobuf whatever the upstream wants. + ### `services.hyperhive.otel.headersCredential` — string or null, default `null` Absolute path to a secret file on the host holding the upstream auth header as `NAME=value` (e.g. `Authorization=Bearer `). -**Only the host collector reads it.** It arrives as an `EnvironmentFile` on the -collector's unit, so the value is never read by nix, never copied into the -store or the generated config, never passed in argv — and **never forwarded -into an agent container**. An agent cannot read the hive's upstream credential -because it is never given one. +**Only the swarm's collector reads it** — the one tier that talks to the +upstream. It arrives as an `EnvironmentFile` on that unit, so the value is +never read by nix, never copied into the store or the generated config, never +passed in argv — and reaches **neither an agent container nor a hive's own +collector**. An agent cannot read the upstream credential because it is never +given one. + +Set it on the host running the swarm's services; a hive that only forwards has +no use for it. Leave `null` if the upstream needs no auth header; the collector then sends none rather than an empty one. @@ -130,46 +153,56 @@ metrics on process exit, so interval tuning is not required for metrics to be exported. A lower value gives more frequent intermediate flushes within long-running turns — cosmetic, not a correctness knob. -## The host collector +## The two collectors -`enable` starts an OpenTelemetry collector on the host. It is not optional and -there is no second path — that is the whole point: +Telemetry crosses two collectors, and which one you configure depends on what +the host is: + +| | runs where | receives from | does | +|---|---|---|---| +| **hive tier** — `otel.enable` | every hive with agents | that hive's agents, on the bridge | forwards to the swarm tier. Holds no credential, picks no destination | +| **swarm tier** — `swarm.otel.enable` | once per swarm | every hive's collector | writes the swarm's store and exports upstream | + +An all-local host runs both, and needs nothing said about the hop between them. ```nix services.hyperhive.otel = { enable = true; endpoint = "https://collector.example.com/otel"; # the upstream - headersCredential = "/run/secrets/otel-headers"; # only the host reads it + headersCredential = "/run/secrets/otel-headers"; # only the swarm tier reads it }; ``` -**Why it isn't a knob.** Exporting straight to `endpoint` means every agent -needs the credential to authenticate — and the harness delivers that token into -the agent's own `~/.claude/settings.json`, a file the agent can read. `0600` -protects it from other containers, not from the agent itself. As long as the -direct path stays *selectable*, that hole stays selectable; an option that can -reintroduce it is a hole with extra steps. +**Why the hive tier isn't optional.** Exporting straight to `endpoint` means +every agent needs the credential to authenticate — and the harness delivers +that token into the agent's own `~/.claude/settings.json`, a file the agent can +read. `0600` protects it from other containers, not from the agent itself. As +long as the direct path stays *selectable*, that hole stays selectable; an +option that can reintroduce it is a hole with extra steps. -**`endpoint` keeps meaning "where telemetry goes upstream."** The collector -does not redefine it — the agent-facing value is *derived* +**Why the tiers stay separate on one box.** They are not collapsed when +co-located: an all-local hive is a statement about *where* processes run, not +about the shape of the deployment. A boundary that disappears locally is one +the local deployment stops testing. + +**`endpoint` keeps meaning "where telemetry goes upstream."** Neither tier +redefines it — the agent-facing value is *derived* (`http://:`), so an existing deployment's `endpoint` keeps working unchanged. The bridge port is contributed to `exposeHostPorts` automatically; there is nothing to open by hand. -What the collector *added* is a second destination: on a host running the -swarm's metrics store it writes there too, so `endpoint` is no longer the only -place telemetry can land — and no longer the only way to have one. - ### `services.hyperhive.otel.collector.port` — port, default `4318` -The OTLP/HTTP port the collector listens on, bound to the bridge IP only. +The OTLP/HTTP port the hive tier listens on, bound to the bridge IP only. The +swarm tier has its own (`swarm.otel.port`, default `4319`) — they share a +network namespace when co-located, so the two must differ. ### `services.hyperhive.otel.collector.upstreamHeaderName` — string, default `"Authorization"` -Name of the header the collector sends upstream. The **value** comes from the +Name of the header the swarm tier sends upstream. The **value** comes from the credential file at runtime (`EnvironmentFile` → `${env:}`), never from nix — so header names are config and header values are secrets, which is the -only split the collector's static header map can express. +only split a static header map can express. ⚠️ **`endpoint` must be valid for `protocol`.** The upstream exporter follows `otel.protocol` (`grpc` → the gRPC exporter, otherwise OTLP/HTTP), and the gRPC diff --git a/docs/swarm/services.md b/docs/swarm/services.md index 8608cb6c..23742b27 100644 --- a/docs/swarm/services.md +++ b/docs/swarm/services.md @@ -93,15 +93,35 @@ login form is switched off whenever SSO is configured. If you enable Grafana on a host with no authelia, the form stays on and Grafana's default `admin`/`admin` applies; change it before exposing that host. -**Where the data comes from.** With `otel.enable` on, the hive's OTEL -collector writes into this store as well as to any upstream endpoint — -both, not one or the other, since a local store is for looking at this -swarm and an upstream is for whoever aggregates across swarms. That also -means `otel.endpoint` is no longer required when the store runs here: a -hive with a local store already has somewhere for telemetry to go. See -[`../observability.md`](../observability.md). +**Where the data comes from.** The swarm's OTEL collector, below. Neither container is reachable except through the gateway: both bind loopback, and VictoriaMetrics' write endpoint takes no credential, so the collector is the only intended writer. +### Telemetry collector (OTEL) + +The swarm's collector receives from every hive's own collector and is the +only process that decides where telemetry goes: it writes the store above +and exports to `otel.endpoint`, doing both when both are configured. It +also holds the upstream credential, which is why no hive and no agent +needs one. + +It follows `swarm.enableRequiredServices` like the services above, in a +`swarm-otel` container. Its `swarm.otel.port` defaults to `4319` rather +than OTLP's usual `4318`, which the hive tier already uses — swarm +containers share the host's network namespace, so two collectors on one +port is a coin toss at runtime rather than an error at build time. + +| Option | When you'd touch it | +|---|---| +| `swarm.otel.url` | **On every hive that does not run the swarm's services.** It defaults to this host, so a hive left at the default forwards into nothing and loses its telemetry silently. Point it at the services host: `"http://services-host.example:4319"`. | +| `swarm.otel.port` | Only if something else on the services host already claims `4319`. | + +With neither `otel.endpoint` nor the store enabled, this collector is +refused at eval — a tier that receives samples and drops them looks +healthy while losing data. + +Agent-side configuration, and what a hive's own collector does, are in +[`../observability.md`](../observability.md). +