The hand-maintained per-option reference (services.hyperhive.otel.*)
restated what nixosOptionsDoc already generates from the same nix
module's mkOption descriptions into /options/ — two copies of the same
fact, one of which the build keeps honest and the other doesn't.
Trimmed to a pointer, keeping only the architecture/security prose a
flat per-option listing can't express. The port-conflict cross-
reference (hive tier 4318 vs swarm tier 4319) that isn't in either
option's own description is already covered in swarm/services.md,
which this doc already links to.
mara, reviewing this PR: "hives always require an identity, swarm controller
and auth is not optional."
So `requireHiveIdentity` is gone rather than defaulted, and with it every
branch that had to describe an unauthenticated collector. The swarm tier now
serves per-hive receivers only, and `/` answers 404 because there is no
swarm-wide inbox to route to. A hive with no credential is a build error, not
a quieter mode.
`hivePortBase` goes too: with per-hive receivers unconditional, `port` IS the
base of the range. That keeps one documented knob instead of adding a second,
and its advice ("move it if something else claims that range") still holds.
Two assertions replace the toggle — an empty hive roster, and a null
`authelia.url`. The second matters because a guessed issuer URL evaluates
cleanly, deploys cleanly, and then refuses every hive at runtime.
⚠️ `cfg.port` is deliberately no longer compared against the derived range in
the collision assertion: it is now the range's first element, so listing it
would make that assertion fire on every config.
This also retires the asymmetry guard added earlier in review — the state it
protected against (auth off on one side, credential still set on the other)
is no longer representable.
Turning ingest auth off without clearing a hive's credential leaves that
hive's collector authenticating and addressing its own path, while an
unauthenticated swarm tier serves one catch-all and forwards the URI
unchanged. The receiver is asked for a path it does not serve, so telemetry
stops with 404s and retries — no 401, no assertion, nothing in any log
naming auth.
Only reachable by overriding one side without the other, since both defaults
derive from the same flag. That is what makes it worth a build error rather
than a caveat: an operator who flips the documented escape hatch has no
reason to suspect the sending half.
Found in review by argus.
Three readers, three different questions:
`observability.md` gets the operator's version — what to set (nothing, on an
all-local swarm), what to set elsewhere, what a misconfiguration looks like
(401s and a missing hive), and what turning it off actually costs.
`swarm/secrets.md` extends the existing client-secret row rather than adding
one, because this is a new *reader* of a secret already described. It notes the
one thing that makes this reader unusual: a host unit reads the file in place,
so unlike every other consumer there is no delivery step to get wrong.
`setup.md` gets one bullet under security notes, since a first-run hive gets
the property without doing anything and should know it has it.
Review feedback from mara ("does the otel info even belong in there?") and
argus: `docs/observability.md` already states twice that the collector's
bridge port is contributed automatically and that neither line is needed for
hyperhive's own telemetry. A second copy of that reasoning in the option
description is a third place to drift from — and drift is what produced the
misleading example this PR started out fixing.
So remove the explanation rather than relocate it. With no otel content in
the option at all there is nothing to keep in sync, and observability.md
stays the one place that owns the otel-specific reasoning.
`example = [ 4318 ]` goes too: 4318 is the OTLP port, so the example still
pointed at otel without naming it.
Also fixes the same confusion one doc over, since it is this PR's subject:
observability.md's own "Network access" section used `otel.endpoint` as the
example for pointing something *else* at a host-local service, immediately
above a warning that you do not need it for otel. It now uses a generic port
and names the reason loopback is wrong from inside a container. While there,
the bridge is `hive-br0`, not `hvbr0` — the only occurrence in the tree.
Refs #3402
hive-c0re's container-resource exporter already targets this hive's own
collector (environment.nix derives the bridge address), so the upstream
header it was loaded with has nowhere to be presented: that hop is
unauthenticated for every producer on the host, and the credential
belongs to the swarm tier, which is the one that leaves the swarm.
Drop the LoadCredential entry and the auth_headers() reader with it.
The option itself stays -- swarm-otel.nix is its real consumer, via
EnvironmentFile on the collector unit.
Also corrects three descriptions that this makes false, or that were
already false: the module doc claimed to reuse the config "Claude Code's
in-container SDK export uses", which stopped being true when agents
moved off that path; the nix comment claimed the secret is "the same one
the agent containers get, forwarded via nspawn --load-credential", which
lost its last producer earlier; and docs/observability.md described an
Authorization header on a hop that will no longer send one. The
headersCredential option's own docs already said it reaches "neither an
agent container nor a hive's own collector" -- this makes that true
rather than aspirational.
The endpoint, protocol, temporality preference and resource labels were
shipped only inside claude's managed settings json, so they applied to
claude's own process. hive-bash-daemon, hive-mcp-http, hive-matrix-daemon
and hive-forge-notify are systemd *siblings* of claude rather than its
children, so nothing shipped there could ever reach them: `hive-metric`
invoked from a tool call exited with "OTEL_EXPORTER_OTLP_ENDPOINT not
set", which is the honest failure of a value it structurally could not
see.
Declare those variables container-wide in a new agent module instead —
systemd.globalEnvironment for every unit PID 1 starts, environment.variables
for login shells. Both are needed and neither implies the other; NIX_REMOTE
is set both ways for the same reason.
Claude keeps only what is genuinely its own: the telemetry master flag,
the feedback-survey flag, the version label, and which signals it
exports. A different producer in the same container may legitimately
emit only metrics.
The hyperhive.otel.* options move across with them. They have more than
one consumer now, so their home is the OTEL module rather than the
claude one.
Drops swarm.otel.url (a loopback default an operator had to override on a
split host) in favor of swarm.otel.domain -- the same
gateway.localNames + nginx-vhost-through-the-gateway shape every other
swarm service (authelia, grafana, victoriametrics, ui) already uses. The
hive tier's exporter now reaches it as https://<domain> unconditionally,
resolved locally by dnsmasq on a co-located host and over the real network
otherwise, instead of a config knob nobody sets until they hit the silent
drop.
Costs CA trust on the hive tier: otel.nix wires
lib/hive-ca-trust.nix's trustBundle with hostUnit = true on the
opentelemetry-collector host unit, the same flag #3441/#3442 added for
swarm-controller and hive-c0re.
mara, #3125 comment 58363: "go c".
observability.md described a single collector holding the upstream
credential. It also said endpoint and protocol are what agents are
handed; agents get the derived first hop, which has been true since the
collector was introduced.
The swarm tier is documented beside its sibling swarm services rather
than here, and the one line an operator must not miss - swarm.otel.url
on a hive that does not run them - is called out in both places, since
leaving it unset loses telemetry silently.
Review catch: this PR relaxed the `otel.endpoint` assertion and staled the
canonical OTEL reference in the same stroke — `docs/observability.md` is
what CLAUDE.md points readers at for "what OTEL options are available",
and it still said required-full-stop while the new swarm/services.md
section said a local store satisfies it.
Also corrects the option's own description in otel.nix, which said the
same thing and renders into the generated options doc. Grepping the
reviewer's phrasing did not find that one; grepping the claim did.
Records the second destination where the "endpoint is where telemetry
ultimately goes" paragraph makes its claim, rather than only in the new
section a reader may not reach.
argus on #3280: the receiver has no auth extension - the nixpkgs module
passes settings straight through and nothing wires one on - so
'unauthenticated to a bridge address' means reachable from an agent
container, not presents a credential. The firewall is the whole access
control.
Consequence, stated because it is a choice rather than an oversight:
any agent can push arbitrary OTLP and the collector forwards it under
the operator's credential, including data smuggled out in resource
attributes. That is a different risk from the one the collector fixes,
and strictly smaller than what preceded it - before, every agent held
the credential itself and could do all of that plus use the token
anywhere else. The collector removes the token and keeps the pipe.
Same principle this PR already applies to the availability trade: state
it where the reader is, rather than let it be discovered.
The headersCredential section still said hive-c0re forwards the file
into each agent container - that is exactly the behaviour this change
removes, so the doc contradicted the code rather than merely lagging
it. Same for the collector section, which presented the two paths the
change collapses.
Also drops the dangling collector.enable reference in the network
section.
argus's non-blocking note on #3278: docs/observability.md is what this
repo's reading-paths index names as 'what OTEL options are available',
and it did not mention collector.enable/port/upstreamHeaderName at all.
The nix docstrings covered it, but not where a reader following the
established path would look.
Carries the two things a docstring is a poor home for: that endpoint
keeps meaning 'where telemetry ultimately goes' (the agent-facing value
is derived, so an existing deployment is unaffected), and the
availability trade the collector makes against the direct-export
property this page already promises.
Two sites spelled it as a brace group, services.hyperhive.{hiveName,
swarmName}, which no anchored rewrite can handle correctly now that only
one of the two moves; both are written out separately. One of them is an
MCP tool description, so it is rendered into every agent's system prompt.
hive-c0re already samples each agent container's cgroup load for the
dashboard (stats/container_stats.rs); this rides those gauges out to the
configured OTLP endpoint, reusing the existing services.hyperhive.otel
config (endpoint + auth header) — no new toggle.
- New stats/otel_metrics.rs: exports via the OpenTelemetry Rust SDK (same
crates as hive-metric) with the semconv container.* metric names +
container.name attribute so off-the-shelf OTel/Grafana dashboards work,
plus the hive agent label. container.cpu.time (counter, s, from cumulative
cpu.stat usage_usec), container.memory.usage, container.memory.usage.limit;
memory peak / on-disk storage / instantaneous cpu percent stay hyperhive.*
custom (no semconv equivalent). Observable instruments read a shared
snapshot an async task refreshes (gather() is async; SDK callbacks sync).
- container_stats: expose cpu_time_usec (cumulative) on ContainerResource.
- The OTLP auth header is loaded onto hive-c0re's own unit via systemd
LoadCredential and read from $CREDENTIALS_DIRECTORY/otel-headers.
- docs/observability.md documents the host-emitted semconv metrics.
Host-side export, so it covers containers even when their agent is idle.
Add docs/observability.md covering all services.hyperhive.otel.*
options: enable, endpoint, protocol, headersCredential,
extraResourceAttributes, debug (new in cb0a66147a), and
metricIntervalMs.
Includes:
- Built-in OTEL_RESOURCE_ATTRIBUTES labels (service.name, agent, hive, swarm)
- Cumulative temporality note (avoids Prometheus DELTA drop)
- Network note for host-side collectors on non-standard ports,
cross-referencing docs/network.md exposeHostPorts
Also:
- CLAUDE.md: add reading-path entry for the new doc
- docs/network.md: link the OTEL mention to observability.md
Closes no issue — gap found during doc sweep.
