docs(#3125): the collector pair, and what an operator sets on which host
observability.md described a single collector holding the upstream credential. It also said endpoint and protocol are what agents are handed; agents get the derived first hop, which has been true since the collector was introduced. The swarm tier is documented beside its sibling swarm services rather than here, and the one line an operator must not miss - swarm.otel.url on a hive that does not run them - is called out in both places, since leaving it unset loses telemetry silently.
This commit is contained in:
atlas
mara
parent
094a54e785
commit
80c9118f87
3 changed files with 106 additions and 50 deletions
|
|
@ -177,8 +177,11 @@ namespace.
|
|||
### Reaching host services (`exposeHostPorts`)
|
||||
|
||||
By default agents can only reach the host on 80/443 (+53 DNS), so a
|
||||
host-side service on another port — e.g. a dev OTEL collector for
|
||||
`services.hyperhive.otel.endpoint` (see `docs/observability.md`) — is unreachable.
|
||||
host-side service on another port — e.g. a dev OTLP collector you want
|
||||
agents to reach directly — is unreachable. (hyperhive's own telemetry
|
||||
needs none of this: `otel.enable` opens its collector's port itself, and
|
||||
`otel.endpoint` is the *upstream*, which no agent ever dials. See
|
||||
`docs/observability.md`.)
|
||||
|
||||
`services.hyperhive.network.exposeHostPorts = [ 4318 ];` opens each
|
||||
listed TCP port `P` on the bridge-interface `allowedTCPPorts`, so an
|
||||
|
|
|
|||
|
|
@ -16,18 +16,30 @@ services.hyperhive.otel = {
|
|||
};
|
||||
```
|
||||
|
||||
`enable` is the single gate. `endpoint` is where telemetry goes upstream —
|
||||
required when enabled *unless* this host runs the swarm's own metrics store
|
||||
(`swarm.victoriametrics.enable`), which is a destination in its own right. With
|
||||
both, telemetry goes to both. See
|
||||
`enable` is the single gate. `endpoint` is where telemetry ends up after it
|
||||
leaves the swarm — optional, because the swarm's own metrics store
|
||||
(`swarm.victoriametrics.enable`) is a destination in its own right. With both,
|
||||
telemetry goes to both. See
|
||||
[`swarm/services.md`](swarm/services.md#metrics-victoriametrics--grafana).
|
||||
|
||||
**There is exactly one way telemetry leaves a hive: through the collector that
|
||||
`enable` starts on the host.** Agents never talk to `endpoint` themselves —
|
||||
they export unauthenticated to a bridge address only their own containers can
|
||||
reach, and the collector forwards upstream with the auth header. So the
|
||||
upstream credential exists in one place, on the host, and no agent ever holds
|
||||
a copy.
|
||||
reach. That collector forwards to the swarm's
|
||||
([`swarm/services.md`](swarm/services.md#telemetry-collector-otel)), which is
|
||||
the single process holding the upstream credential and the only writer to the
|
||||
swarm's store. No agent holds a copy, and neither does this hive.
|
||||
|
||||
⚠️ **On a hive that does not run the swarm's services, say where that swarm
|
||||
collector is:**
|
||||
|
||||
```nix
|
||||
services.hyperhive.swarm.otel.url = "http://services-host.example:4319";
|
||||
```
|
||||
|
||||
Left unset it points at this host, where nothing is listening — the collector
|
||||
starts, agents export happily, and the samples go nowhere. The service host
|
||||
itself needs no such line.
|
||||
|
||||
⚠️ **The collector is therefore in the path of all telemetry.** It runs on the
|
||||
same host as the agents and restarts on failure, and telemetry is not the
|
||||
|
|
@ -42,8 +54,8 @@ port on the bridge interface only. So "unauthenticated to a bridge address"
|
|||
means *reachable from an agent container*, not *presents a credential*.
|
||||
|
||||
The consequence, stated because it is a choice rather than an oversight: **any
|
||||
agent can push arbitrary OTLP, and the collector forwards it upstream under the
|
||||
operator's credential.** It cannot tell a container's genuine Claude Code stats
|
||||
agent can push arbitrary OTLP, and it is forwarded on under the operator's
|
||||
credential.** Neither tier can tell a container's genuine Claude Code stats
|
||||
from anything else shaped like OTLP arriving on that port — including data
|
||||
smuggled out in resource attributes on an otherwise-legitimate export.
|
||||
|
||||
|
|
@ -61,33 +73,44 @@ closed by this design, and nothing here should be read as closing it.
|
|||
|
||||
Master switch. When true, all other options below take effect.
|
||||
|
||||
### `services.hyperhive.otel.endpoint` — string, required when enabled unless the swarm store runs here
|
||||
### `services.hyperhive.otel.endpoint` — string, default `""`
|
||||
|
||||
Upstream OTLP endpoint URL. Set as `OTEL_EXPORTER_OTLP_ENDPOINT` for every
|
||||
agent. Example: `"https://collector.example.com/otel"`.
|
||||
Upstream OTLP endpoint URL, read by the swarm's collector. Example:
|
||||
`"https://collector.example.com/otel"`.
|
||||
|
||||
Leave it empty **only** on a host running `swarm.victoriametrics.enable` — the
|
||||
local store is then the destination and the collector writes there instead.
|
||||
With neither, `enable` is refused at eval: telemetry with nowhere to go is a
|
||||
misconfiguration, not a quiet no-op.
|
||||
Leave it empty on a swarm running its own metrics store — that store is then
|
||||
the destination. With neither, the swarm collector is refused at eval:
|
||||
telemetry with nowhere to go is a misconfiguration, not a quiet no-op.
|
||||
|
||||
Not what agents are handed. Their endpoint is this hive's own collector,
|
||||
derived from the bridge address, so setting this changes where telemetry
|
||||
*ends up* and never what a container is told.
|
||||
|
||||
### `services.hyperhive.otel.protocol` — enum, default `"http/protobuf"`
|
||||
|
||||
OTLP wire protocol, passed as `OTEL_EXPORTER_OTLP_PROTOCOL`. Accepted values:
|
||||
Wire protocol for the **upstream** link, honoured by the swarm collector's
|
||||
exporter. Accepted values:
|
||||
- `"http/protobuf"` (default)
|
||||
- `"http/json"`
|
||||
- `"grpc"`
|
||||
|
||||
Agents are not affected: their first hop is this hive's collector, whose
|
||||
OTLP/HTTP receiver takes protobuf whatever the upstream wants.
|
||||
|
||||
### `services.hyperhive.otel.headersCredential` — string or null, default `null`
|
||||
|
||||
Absolute path to a secret file on the host holding the upstream auth header as
|
||||
`NAME=value` (e.g. `Authorization=Bearer <token>`).
|
||||
|
||||
**Only the host collector reads it.** It arrives as an `EnvironmentFile` on the
|
||||
collector's unit, so the value is never read by nix, never copied into the
|
||||
store or the generated config, never passed in argv — and **never forwarded
|
||||
into an agent container**. An agent cannot read the hive's upstream credential
|
||||
because it is never given one.
|
||||
**Only the swarm's collector reads it** — the one tier that talks to the
|
||||
upstream. It arrives as an `EnvironmentFile` on that unit, so the value is
|
||||
never read by nix, never copied into the store or the generated config, never
|
||||
passed in argv — and reaches **neither an agent container nor a hive's own
|
||||
collector**. An agent cannot read the upstream credential because it is never
|
||||
given one.
|
||||
|
||||
Set it on the host running the swarm's services; a hive that only forwards has
|
||||
no use for it.
|
||||
|
||||
Leave `null` if the upstream needs no auth header; the collector then sends
|
||||
none rather than an empty one.
|
||||
|
|
@ -130,46 +153,56 @@ metrics on process exit, so interval tuning is not required for metrics to be
|
|||
exported. A lower value gives more frequent intermediate flushes within
|
||||
long-running turns — cosmetic, not a correctness knob.
|
||||
|
||||
## The host collector
|
||||
## The two collectors
|
||||
|
||||
`enable` starts an OpenTelemetry collector on the host. It is not optional and
|
||||
there is no second path — that is the whole point:
|
||||
Telemetry crosses two collectors, and which one you configure depends on what
|
||||
the host is:
|
||||
|
||||
| | runs where | receives from | does |
|
||||
|---|---|---|---|
|
||||
| **hive tier** — `otel.enable` | every hive with agents | that hive's agents, on the bridge | forwards to the swarm tier. Holds no credential, picks no destination |
|
||||
| **swarm tier** — `swarm.otel.enable` | once per swarm | every hive's collector | writes the swarm's store and exports upstream |
|
||||
|
||||
An all-local host runs both, and needs nothing said about the hop between them.
|
||||
|
||||
```nix
|
||||
services.hyperhive.otel = {
|
||||
enable = true;
|
||||
endpoint = "https://collector.example.com/otel"; # the upstream
|
||||
headersCredential = "/run/secrets/otel-headers"; # only the host reads it
|
||||
headersCredential = "/run/secrets/otel-headers"; # only the swarm tier reads it
|
||||
};
|
||||
```
|
||||
|
||||
**Why it isn't a knob.** Exporting straight to `endpoint` means every agent
|
||||
needs the credential to authenticate — and the harness delivers that token into
|
||||
the agent's own `~/.claude/settings.json`, a file the agent can read. `0600`
|
||||
protects it from other containers, not from the agent itself. As long as the
|
||||
direct path stays *selectable*, that hole stays selectable; an option that can
|
||||
reintroduce it is a hole with extra steps.
|
||||
**Why the hive tier isn't optional.** Exporting straight to `endpoint` means
|
||||
every agent needs the credential to authenticate — and the harness delivers
|
||||
that token into the agent's own `~/.claude/settings.json`, a file the agent can
|
||||
read. `0600` protects it from other containers, not from the agent itself. As
|
||||
long as the direct path stays *selectable*, that hole stays selectable; an
|
||||
option that can reintroduce it is a hole with extra steps.
|
||||
|
||||
**`endpoint` keeps meaning "where telemetry goes upstream."** The collector
|
||||
does not redefine it — the agent-facing value is *derived*
|
||||
**Why the tiers stay separate on one box.** They are not collapsed when
|
||||
co-located: an all-local hive is a statement about *where* processes run, not
|
||||
about the shape of the deployment. A boundary that disappears locally is one
|
||||
the local deployment stops testing.
|
||||
|
||||
**`endpoint` keeps meaning "where telemetry goes upstream."** Neither tier
|
||||
redefines it — the agent-facing value is *derived*
|
||||
(`http://<bridgeIp>:<collector.port>`), so an existing deployment's `endpoint`
|
||||
keeps working unchanged. The bridge port is contributed to `exposeHostPorts`
|
||||
automatically; there is nothing to open by hand.
|
||||
|
||||
What the collector *added* is a second destination: on a host running the
|
||||
swarm's metrics store it writes there too, so `endpoint` is no longer the only
|
||||
place telemetry can land — and no longer the only way to have one.
|
||||
|
||||
### `services.hyperhive.otel.collector.port` — port, default `4318`
|
||||
|
||||
The OTLP/HTTP port the collector listens on, bound to the bridge IP only.
|
||||
The OTLP/HTTP port the hive tier listens on, bound to the bridge IP only. The
|
||||
swarm tier has its own (`swarm.otel.port`, default `4319`) — they share a
|
||||
network namespace when co-located, so the two must differ.
|
||||
|
||||
### `services.hyperhive.otel.collector.upstreamHeaderName` — string, default `"Authorization"`
|
||||
|
||||
Name of the header the collector sends upstream. The **value** comes from the
|
||||
Name of the header the swarm tier sends upstream. The **value** comes from the
|
||||
credential file at runtime (`EnvironmentFile` → `${env:<name>}`), never from
|
||||
nix — so header names are config and header values are secrets, which is the
|
||||
only split the collector's static header map can express.
|
||||
only split a static header map can express.
|
||||
|
||||
⚠️ **`endpoint` must be valid for `protocol`.** The upstream exporter follows
|
||||
`otel.protocol` (`grpc` → the gRPC exporter, otherwise OTLP/HTTP), and the gRPC
|
||||
|
|
|
|||
|
|
@ -93,15 +93,35 @@ login form is switched off whenever SSO is configured. If you enable
|
|||
Grafana on a host with no authelia, the form stays on and Grafana's
|
||||
default `admin`/`admin` applies; change it before exposing that host.
|
||||
|
||||
**Where the data comes from.** With `otel.enable` on, the hive's OTEL
|
||||
collector writes into this store as well as to any upstream endpoint —
|
||||
both, not one or the other, since a local store is for looking at this
|
||||
swarm and an upstream is for whoever aggregates across swarms. That also
|
||||
means `otel.endpoint` is no longer required when the store runs here: a
|
||||
hive with a local store already has somewhere for telemetry to go. See
|
||||
[`../observability.md`](../observability.md).
|
||||
**Where the data comes from.** The swarm's OTEL collector, below.
|
||||
|
||||
Neither container is reachable except through the gateway: both bind
|
||||
loopback, and VictoriaMetrics' write endpoint takes no credential, so
|
||||
the collector is the only intended writer.
|
||||
|
||||
### Telemetry collector (OTEL)
|
||||
|
||||
The swarm's collector receives from every hive's own collector and is the
|
||||
only process that decides where telemetry goes: it writes the store above
|
||||
and exports to `otel.endpoint`, doing both when both are configured. It
|
||||
also holds the upstream credential, which is why no hive and no agent
|
||||
needs one.
|
||||
|
||||
It follows `swarm.enableRequiredServices` like the services above, in a
|
||||
`swarm-otel` container. Its `swarm.otel.port` defaults to `4319` rather
|
||||
than OTLP's usual `4318`, which the hive tier already uses — swarm
|
||||
containers share the host's network namespace, so two collectors on one
|
||||
port is a coin toss at runtime rather than an error at build time.
|
||||
|
||||
| Option | When you'd touch it |
|
||||
|---|---|
|
||||
| `swarm.otel.url` | **On every hive that does not run the swarm's services.** It defaults to this host, so a hive left at the default forwards into nothing and loses its telemetry silently. Point it at the services host: `"http://services-host.example:4319"`. |
|
||||
| `swarm.otel.port` | Only if something else on the services host already claims `4319`. |
|
||||
|
||||
With neither `otel.endpoint` nor the store enabled, this collector is
|
||||
refused at eval — a tier that receives samples and drops them looks
|
||||
healthy while losing data.
|
||||
|
||||
Agent-side configuration, and what a hive's own collector does, are in
|
||||
[`../observability.md`](../observability.md).
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue