vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

docs(otel): one path, and stop describing a forwarding that no longer happens

Browse source 
The headersCredential section still said hive-c0re forwards the file
into each agent container - that is exactly the behaviour this change
removes, so the doc contradicted the code rather than merely lagging
it. Same for the collector section, which presented the two paths the
change collapses.

Also drops the dangling collector.enable reference in the network
section.
This commit is contained in:
atlas 2026-08-15 10:22:12 +02:00 • committed by mara
commit 513eb5729d
2 changed files with 54 additions and 56 deletions
Download patch file Download diff file Expand all files Collapse all files

87
docs/observability.md
View file

@ -16,12 +16,20 @@ services.hyperhive.otel = {
}; };
``` ```
`enable` is the single gate. `endpoint` (required when enabled) is the OTLP `enable` is the single gate. `endpoint` (required when enabled) is where
HTTP endpoint; hive-c0re injects it as `OTEL_EXPORTER_OTLP_ENDPOINT` into telemetry ultimately goes.
every agent's systemd service via the generated meta flake.
Each agent's harness (hive-ag3nt) exports directly to the collector — the **There is exactly one way telemetry leaves a hive: through the collector that
pipeline keeps working even when hive-c0re is down. `enable` starts on the host.** Agents never talk to `endpoint` themselves —
they export unauthenticated to a bridge address only their own containers can
reach, and the collector forwards upstream with the auth header. So the
upstream credential exists in one place, on the host, and no agent ever holds
a copy.
⚠️ **The collector is therefore in the path of all telemetry.** It runs on the
same host as the agents and restarts on failure, and telemetry is not the
control plane — degraded telemetry is not degraded operation — but the export
no longer survives independently of anything host-side.
## Options reference ## Options reference
@ -43,17 +51,17 @@ OTLP wire protocol, passed as `OTEL_EXPORTER_OTLP_PROTOCOL`. Accepted values:
### `services.hyperhive.otel.headersCredential` — string or null, default `null` ### `services.hyperhive.otel.headersCredential` — string or null, default `null`
Absolute path to a secret file on the host whose contents become Absolute path to a secret file on the host holding the upstream auth header as
`OTEL_EXPORTER_OTLP_HEADERS` (e.g. `Authorization=Bearer <token>`). `NAME=value` (e.g. `Authorization=Bearer <token>`).
hive-c0re forwards this host file into each agent container via **Only the host collector reads it.** It arrives as an `EnvironmentFile` on the
`systemd-nspawn --load-credential=otel-headers:<path>`; the inner harness unit collector's unit, so the value is never read by nix, never copied into the
inherits it by name. The token is never copied into the nix store, generated store or the generated config, never passed in argv — and **never forwarded
config, a bind mount, or argv. into an agent container**. An agent cannot read the hive's upstream credential
because it is never given one.
Leave `null` if the endpoint needs no auth header. A configured-but-missing Leave `null` if the upstream needs no auth header; the collector then sends
file is skipped with a log warning — OTEL still exports, just without the auth none rather than an empty one.
header.
```nix ```nix
services.hyperhive.otel = { services.hyperhive.otel = {
@ -93,44 +101,31 @@ metrics on process exit, so interval tuning is not required for metrics to be
exported. A lower value gives more frequent intermediate flushes within exported. A lower value gives more frequent intermediate flushes within
long-running turns — cosmetic, not a correctness knob. long-running turns — cosmetic, not a correctness knob.
## Running a collector on the host (`otel.collector`) ## The host collector
By default every agent exports **straight to `endpoint`**, which means every `enable` starts an OpenTelemetry collector on the host. It is not optional and
agent needs `headersCredential` to authenticate — and the harness delivers that there is no second path — that is the whole point:
token into the agent's own `~/.claude/settings.json`, a file the agent can read.
`0600` protects it from other containers, not from the agent itself.
With `services.hyperhive.otel.collector.enable = true`, a collector runs on the
host and holds the credential instead. Agents export **unauthenticated** to a
bridge address only their own containers can reach; the collector adds the
upstream header and forwards to `endpoint`.
```nix ```nix
services.hyperhive.otel = { services.hyperhive.otel = {
enable = true; enable = true;
endpoint = "https://collector.example.com/otel"; # still the upstream endpoint = "https://collector.example.com/otel"; # the upstream
headersCredential = "/run/secrets/otel-headers"; # now only the host reads it headersCredential = "/run/secrets/otel-headers"; # only the host reads it
collector.enable = true;
}; };
``` ```
**`endpoint` keeps meaning "where telemetry ultimately goes."** Turning the **Why it isn't a knob.** Exporting straight to `endpoint` means every agent
collector on does not redefine it — the agent-facing value is *derived* needs the credential to authenticate — and the harness delivers that token into
the agent's own `~/.claude/settings.json`, a file the agent can read. `0600`
protects it from other containers, not from the agent itself. As long as the
direct path stays *selectable*, that hole stays selectable; an option that can
reintroduce it is a hole with extra steps.
**`endpoint` keeps meaning "where telemetry ultimately goes."** The collector
does not redefine it — the agent-facing value is *derived*
(`http://<bridgeIp>:<collector.port>`), so an existing deployment's `endpoint` (`http://<bridgeIp>:<collector.port>`), so an existing deployment's `endpoint`
keeps working unchanged. The bridge port is contributed to keeps working unchanged. The bridge port is contributed to `exposeHostPorts`
`exposeHostPorts` automatically; there is nothing to open by hand. automatically; there is nothing to open by hand.
⚠️ **The trade:** without a collector each harness exports directly, so
telemetry survives anything host-side being down. A local collector is a new
dependency in that path. It runs on the same host as the agents, so the window
is small — but it is not zero.
### `services.hyperhive.otel.collector.enable` — bool, default `false`
Off means *absent*: no unit, no port, and `endpoint` keeps its current meaning
for every agent. Requires `headersCredential` to be set — a collector with no
credential is pure indirection, and an assertion says so rather than letting it
deploy.
### `services.hyperhive.otel.collector.port` — port, default `4318` ### `services.hyperhive.otel.collector.port` — port, default `4318`
@ -169,9 +164,9 @@ services.hyperhive.otel.endpoint = "http://10.42.0.1:4318";
The bridge IP is the host's address on the `hvbr0` bridge, typically The bridge IP is the host's address on the `hvbr0` bridge, typically
`10.42.0.1`. See `docs/network.md::Reaching host services` for details. `10.42.0.1`. See `docs/network.md::Reaching host services` for details.
This is the manual form of what `otel.collector.enable` does for you — with the ⚠️ **You do not need either line for hyperhive's own telemetry** — `otel.enable`
collector on, the port is contributed and the endpoint derived, so neither line contributes the collector's port and derives the endpoint itself. The above is
above is needed. for pointing something *else* at a host-local service.
## Built-in resource labels ## Built-in resource labels

23
nix/host-modules/otel.nix
View file

@ -62,16 +62,19 @@
default = null; default = null;
example = "/run/secrets/otel-headers"; example = "/run/secrets/otel-headers";
description = '' description = ''
Absolute path to an operator-provided secret file whose contents Absolute path to an operator-provided secret file holding the
become `OTEL_EXPORTER_OTLP_HEADERS` (e.g. upstream auth header as `NAME=value` (e.g.
`Authorization=Bearer <token>`). hive-c0re forwards this host `Authorization=Bearer <token>`).
file into each agent container's credential store via
systemd-nspawn `--load-credential=otel-headers:<path>`; the inner **Only the host-side collector reads this.** It reaches the
harness unit inherits it by name (`LoadCredential`), so the token collector as an `EnvironmentFile`, so the value is never read by
is never copied into the nix store, the generated config, a bind nix, never copied into the store or the generated config, and
mount, or argv. Must be absolute. Leave null if the endpoint never passed in argv — and it is never forwarded into an agent
needs no auth header. A configured-but-missing file is skipped container, which is the point of the collector existing. Must be
with a log warning (OTEL still exports, without the auth header). absolute.
Leave null if the upstream needs no auth header; the collector
then sends none rather than an empty one.
''; '';
}; };