vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(#3125): the collector pair, and what an operator sets on which host

Browse source 
observability.md described a single collector holding the upstream
credential. It also said endpoint and protocol are what agents are
handed; agents get the derived first hop, which has been true since the
collector was introduced.

The swarm tier is documented beside its sibling swarm services rather
than here, and the one line an operator must not miss - swarm.otel.url
on a hive that does not run them - is called out in both places, since
leaving it unset loses telemetry silently.
This commit is contained in:
atlas 2026-08-18 13:14:06 +02:00 • committed by mara
commit 80c9118f87
3 changed files with 106 additions and 50 deletions
Download patch file Download diff file Expand all files Collapse all files

115
docs/observability.md
View file

@ -16,18 +16,30 @@ services.hyperhive.otel = {
};
```
`enable` is the single gate. `endpoint` is where telemetry goes upstream —
required when enabled *unless* this host runs the swarm's own metrics store
(`swarm.victoriametrics.enable`), which is a destination in its own right. With
both, telemetry goes to both. See
`enable` is the single gate. `endpoint` is where telemetry ends up after it
leaves the swarm — optional, because the swarm's own metrics store
(`swarm.victoriametrics.enable`) is a destination in its own right. With both,
telemetry goes to both. See
[`swarm/services.md`](swarm/services.md#metrics-victoriametrics--grafana).
**There is exactly one way telemetry leaves a hive: through the collector that
`enable` starts on the host.** Agents never talk to `endpoint` themselves —
they export unauthenticated to a bridge address only their own containers can
reach, and the collector forwards upstream with the auth header. So the
upstream credential exists in one place, on the host, and no agent ever holds
a copy.
reach. That collector forwards to the swarm's
([`swarm/services.md`](swarm/services.md#telemetry-collector-otel)), which is
the single process holding the upstream credential and the only writer to the
swarm's store. No agent holds a copy, and neither does this hive.
⚠️ **On a hive that does not run the swarm's services, say where that swarm
collector is:**
```nix
services.hyperhive.swarm.otel.url = "http://services-host.example:4319";
```
Left unset it points at this host, where nothing is listening — the collector
starts, agents export happily, and the samples go nowhere. The service host
itself needs no such line.
⚠️ **The collector is therefore in the path of all telemetry.** It runs on the
same host as the agents and restarts on failure, and telemetry is not the
@ -42,8 +54,8 @@ port on the bridge interface only. So "unauthenticated to a bridge address"
means *reachable from an agent container*, not *presents a credential*.
The consequence, stated because it is a choice rather than an oversight: **any
agent can push arbitrary OTLP, and the collector forwards it upstream under the
operator's credential.** It cannot tell a container's genuine Claude Code stats
agent can push arbitrary OTLP, and it is forwarded on under the operator's
credential.** Neither tier can tell a container's genuine Claude Code stats
from anything else shaped like OTLP arriving on that port — including data
smuggled out in resource attributes on an otherwise-legitimate export.
@ -61,33 +73,44 @@ closed by this design, and nothing here should be read as closing it.
Master switch. When true, all other options below take effect.
### `services.hyperhive.otel.endpoint` — string, required when enabled unless the swarm store runs here
### `services.hyperhive.otel.endpoint` — string, default `""`
Upstream OTLP endpoint URL. Set as `OTEL_EXPORTER_OTLP_ENDPOINT` for every
agent. Example: `"https://collector.example.com/otel"`.
Upstream OTLP endpoint URL, read by the swarm's collector. Example:
`"https://collector.example.com/otel"`.
Leave it empty **only** on a host running `swarm.victoriametrics.enable` — the
local store is then the destination and the collector writes there instead.
With neither, `enable` is refused at eval: telemetry with nowhere to go is a
misconfiguration, not a quiet no-op.
Leave it empty on a swarm running its own metrics store — that store is then
the destination. With neither, the swarm collector is refused at eval:
telemetry with nowhere to go is a misconfiguration, not a quiet no-op.
Not what agents are handed. Their endpoint is this hive's own collector,
derived from the bridge address, so setting this changes where telemetry
*ends up* and never what a container is told.
### `services.hyperhive.otel.protocol` — enum, default `"http/protobuf"`
OTLP wire protocol, passed as `OTEL_EXPORTER_OTLP_PROTOCOL`. Accepted values:
Wire protocol for the **upstream** link, honoured by the swarm collector's
exporter. Accepted values:
- `"http/protobuf"` (default)
- `"http/json"`
- `"grpc"`
Agents are not affected: their first hop is this hive's collector, whose
OTLP/HTTP receiver takes protobuf whatever the upstream wants.
### `services.hyperhive.otel.headersCredential` — string or null, default `null`
Absolute path to a secret file on the host holding the upstream auth header as
`NAME=value` (e.g. `Authorization=Bearer <token>`).
**Only the host collector reads it.** It arrives as an `EnvironmentFile` on the
collector's unit, so the value is never read by nix, never copied into the
store or the generated config, never passed in argv — and **never forwarded
into an agent container**. An agent cannot read the hive's upstream credential
because it is never given one.
**Only the swarm's collector reads it** — the one tier that talks to the
upstream. It arrives as an `EnvironmentFile` on that unit, so the value is
never read by nix, never copied into the store or the generated config, never
passed in argv — and reaches **neither an agent container nor a hive's own
collector**. An agent cannot read the upstream credential because it is never
given one.
Set it on the host running the swarm's services; a hive that only forwards has
no use for it.
Leave `null` if the upstream needs no auth header; the collector then sends
none rather than an empty one.
@ -130,46 +153,56 @@ metrics on process exit, so interval tuning is not required for metrics to be
exported. A lower value gives more frequent intermediate flushes within
long-running turns — cosmetic, not a correctness knob.
## The host collector
## The two collectors
`enable` starts an OpenTelemetry collector on the host. It is not optional and
there is no second path — that is the whole point:
Telemetry crosses two collectors, and which one you configure depends on what
the host is:
| | runs where | receives from | does |
|---|---|---|---|
| **hive tier** — `otel.enable` | every hive with agents | that hive's agents, on the bridge | forwards to the swarm tier. Holds no credential, picks no destination |
| **swarm tier** — `swarm.otel.enable` | once per swarm | every hive's collector | writes the swarm's store and exports upstream |
An all-local host runs both, and needs nothing said about the hop between them.
```nix
services.hyperhive.otel = {
enable = true;
endpoint = "https://collector.example.com/otel"; # the upstream
headersCredential = "/run/secrets/otel-headers"; # only the host reads it
headersCredential = "/run/secrets/otel-headers"; # only the swarm tier reads it
};
```
**Why it isn't a knob.** Exporting straight to `endpoint` means every agent
needs the credential to authenticate — and the harness delivers that token into
the agent's own `~/.claude/settings.json`, a file the agent can read. `0600`
protects it from other containers, not from the agent itself. As long as the
direct path stays *selectable*, that hole stays selectable; an option that can
reintroduce it is a hole with extra steps.
**Why the hive tier isn't optional.** Exporting straight to `endpoint` means
every agent needs the credential to authenticate — and the harness delivers
that token into the agent's own `~/.claude/settings.json`, a file the agent can
read. `0600` protects it from other containers, not from the agent itself. As
long as the direct path stays *selectable*, that hole stays selectable; an
option that can reintroduce it is a hole with extra steps.
**`endpoint` keeps meaning "where telemetry goes upstream."** The collector
does not redefine it — the agent-facing value is *derived*
**Why the tiers stay separate on one box.** They are not collapsed when
co-located: an all-local hive is a statement about *where* processes run, not
about the shape of the deployment. A boundary that disappears locally is one
the local deployment stops testing.
**`endpoint` keeps meaning "where telemetry goes upstream."** Neither tier
redefines it — the agent-facing value is *derived*
(`http://<bridgeIp>:<collector.port>`), so an existing deployment's `endpoint`
keeps working unchanged. The bridge port is contributed to `exposeHostPorts`
automatically; there is nothing to open by hand.
What the collector *added* is a second destination: on a host running the
swarm's metrics store it writes there too, so `endpoint` is no longer the only
place telemetry can land — and no longer the only way to have one.
### `services.hyperhive.otel.collector.port` — port, default `4318`
The OTLP/HTTP port the collector listens on, bound to the bridge IP only.
The OTLP/HTTP port the hive tier listens on, bound to the bridge IP only. The
swarm tier has its own (`swarm.otel.port`, default `4319`) — they share a
network namespace when co-located, so the two must differ.
### `services.hyperhive.otel.collector.upstreamHeaderName` — string, default `"Authorization"`
Name of the header the collector sends upstream. The **value** comes from the
Name of the header the swarm tier sends upstream. The **value** comes from the
credential file at runtime (`EnvironmentFile` → `${env:<name>}`), never from
nix — so header names are config and header values are secrets, which is the
only split the collector's static header map can express.
only split a static header map can express.
⚠️ **`endpoint` must be valid for `protocol`.** The upstream exporter follows
`otel.protocol` (`grpc` → the gRPC exporter, otherwise OTLP/HTTP), and the gRPC