otel: stop handing hive-c0re the upstream auth credential

hive-c0re's container-resource exporter already targets this hive's own
collector (environment.nix derives the bridge address), so the upstream
header it was loaded with has nowhere to be presented: that hop is
unauthenticated for every producer on the host, and the credential
belongs to the swarm tier, which is the one that leaves the swarm.

Drop the LoadCredential entry and the auth_headers() reader with it.
The option itself stays -- swarm-otel.nix is its real consumer, via
EnvironmentFile on the collector unit.

Also corrects three descriptions that this makes false, or that were
already false: the module doc claimed to reuse the config "Claude Code's
in-container SDK export uses", which stopped being true when agents
moved off that path; the nix comment claimed the secret is "the same one
the agent containers get, forwarded via nspawn --load-credential", which
lost its last producer earlier; and docs/observability.md described an
Authorization header on a hop that will no longer send one. The
headersCredential option's own docs already said it reaches "neither an
agent container nor a hive's own collector" -- this makes that true
rather than aspirational.
This commit is contained in:
atlas 2026-08-18 23:06:14 +02:00 committed by mara
commit 5ca5433e0b
3 changed files with 27 additions and 50 deletions

View file

@ -256,10 +256,9 @@ reference above); custom per-data-point labels can be passed with
When OTEL is enabled, **hive-c0re itself** also exports each agent When OTEL is enabled, **hive-c0re itself** also exports each agent
container's resource load — the same cgroup gauges shown on the dashboard container's resource load — the same cgroup gauges shown on the dashboard
LOAD tab — to the configured `endpoint`, reusing the same LOAD tab — to this hive's own collector, exactly like an agent does and with
`services.hyperhive.otel` config (no separate toggle). These come from the no separate toggle. These come from the host, not the in-container Claude SDK,
host, not the in-container Claude SDK, so they cover containers even when so they cover containers even when their agent is idle.
their agent is idle.
Emitted via the OpenTelemetry Rust SDK, using the Emitted via the OpenTelemetry Rust SDK, using the
[semconv `container.*`](https://opentelemetry.io/docs/specs/semconv/system/container-metrics/) [semconv `container.*`](https://opentelemetry.io/docs/specs/semconv/system/container-metrics/)
@ -284,9 +283,9 @@ labels (`hive`, `swarm`, …) ride on the resource via
`extraResourceAttributes`. `extraResourceAttributes`.
Cadence follows `metricIntervalMs` (default 60s). Transport is OTLP/HTTP Cadence follows `metricIntervalMs` (default 60s). Transport is OTLP/HTTP
(JSON) to `<endpoint>`; the auth header is loaded onto hive-c0re's own unit (JSON) to the hive collector's bridge address, with no auth header — that
via systemd `LoadCredential` (from the same `headersCredential` file) and first hop is unauthenticated for every producer on this host, and the upstream
sent as `Authorization`. credential stays on the swarm tier.
## Agent-emitted custom metrics (`hive-metric`) ## Agent-emitted custom metrics (`hive-metric`)

View file

@ -1,11 +1,11 @@
//! Per-agent container-resource OTEL export. hive-c0re already samples each //! Per-agent container-resource OTEL export. hive-c0re already samples each
//! agent container's cgroup load for the dashboard //! agent container's cgroup load for the dashboard
//! ([`super::container_stats`]); this rides those same gauges out to the //! ([`super::container_stats`]); this rides those same gauges out to the
//! hive-wide OTLP endpoint, reusing the SAME `services.hyperhive.otel` config //! hive's own collector — the same first hop the agents use, arriving as
//! (endpoint + auth header) that Claude Code's in-container SDK export uses — //! `HYPERHIVE_OTEL_ENDPOINT` (see `nix/host-modules/hive-c0re`). No toggle of
//! no new toggle. The auth header is loaded onto hive-c0re's own unit via //! its own, and no credential: a hive's collector takes unauthenticated OTLP
//! systemd `LoadCredential` (see `nix/host-modules/hive-c0re`) and read from //! on the bridge, and the only hop that presents anything is the swarm tier's,
//! `$CREDENTIALS_DIRECTORY/otel-headers`. //! which is the one that leaves the swarm.
//! //!
//! Emits the OTEL **semconv `container.*`** metrics with the standard //! Emits the OTEL **semconv `container.*`** metrics with the standard
//! `container.name` attribute (so off-the-shelf OTEL/Grafana container //! `container.name` attribute (so off-the-shelf OTEL/Grafana container
@ -21,14 +21,13 @@
//! observable-instrument callbacks are sync. So an async task refreshes a //! observable-instrument callbacks are sync. So an async task refreshes a
//! shared snapshot on an interval, and the (sync) callbacks read it. //! shared snapshot on an interval, and the (sync) callbacks read it.
use std::collections::HashMap;
use std::sync::{Arc, Mutex, OnceLock}; use std::sync::{Arc, Mutex, OnceLock};
use std::time::Duration; use std::time::Duration;
use anyhow::{Context, Result}; use anyhow::{Context, Result};
use opentelemetry::KeyValue; use opentelemetry::KeyValue;
use opentelemetry::metrics::MeterProvider as _; use opentelemetry::metrics::MeterProvider as _;
use opentelemetry_otlp::{MetricExporter, Protocol, WithExportConfig, WithHttpConfig}; use opentelemetry_otlp::{MetricExporter, Protocol, WithExportConfig};
use opentelemetry_sdk::Resource; use opentelemetry_sdk::Resource;
use opentelemetry_sdk::metrics::{PeriodicReader, SdkMeterProvider}; use opentelemetry_sdk::metrics::{PeriodicReader, SdkMeterProvider};
@ -90,15 +89,16 @@ fn build_provider(
// http/json — the only OTLP transport this crate enables (matching // http/json — the only OTLP transport this crate enables (matching
// hive-metric). The Claude SDK path honours `HYPERHIVE_OTEL_PROTOCOL` for // hive-metric). The Claude SDK path honours `HYPERHIVE_OTEL_PROTOCOL` for
// its own export; this exporter is always http/json. // its own export; this exporter is always http/json.
let mut builder = MetricExporter::builder() //
// No auth headers: the destination is this hive's own collector, which
// takes unauthenticated OTLP on the bridge. Adding one here would put the
// upstream credential on a hop that never uses it.
let exporter = MetricExporter::builder()
.with_http() .with_http()
.with_endpoint(endpoint) .with_endpoint(endpoint)
.with_protocol(Protocol::HttpJson); .with_protocol(Protocol::HttpJson)
let headers = auth_headers(); .build()
if !headers.is_empty() { .context("build OTLP metric exporter")?;
builder = builder.with_headers(headers);
}
let exporter = builder.build().context("build OTLP metric exporter")?;
// Drive the reader's export at `interval` so `HYPERHIVE_OTEL_METRIC_INTERVAL_MS` // Drive the reader's export at `interval` so `HYPERHIVE_OTEL_METRIC_INTERVAL_MS`
// is the real export cadence (not just the snapshot-refresh cadence). The // is the real export cadence (not just the snapshot-refresh cadence). The
// refresher runs at the same interval, gather-first, so the snapshot is // refresher runs at the same interval, gather-first, so the snapshot is
@ -294,25 +294,6 @@ fn resource_attributes() -> Vec<(String, String)> {
.unwrap_or_default() .unwrap_or_default()
} }
/// OTLP auth headers from the systemd credential at
/// `$CREDENTIALS_DIRECTORY/otel-headers` (an `OTEL_EXPORTER_OTLP_HEADERS`-style
/// `Key=Value` line). Empty when the credential is absent — the collector is
/// then assumed unauthenticated.
fn auth_headers() -> HashMap<String, String> {
let Some(dir) = std::env::var("CREDENTIALS_DIRECTORY").ok() else {
return HashMap::new();
};
let path = std::path::Path::new(&dir).join("otel-headers");
let Ok(raw) = std::fs::read_to_string(&path) else {
tracing::warn!(
"otel container-metrics: no auth header at $CREDENTIALS_DIRECTORY/otel-headers; \
exporting without Authorization"
);
return HashMap::new();
};
parse_kv(&raw).into_iter().collect()
}
/// Parse `key=value` pairs separated by commas and/or newlines. The value /// Parse `key=value` pairs separated by commas and/or newlines. The value
/// keeps any `=` after the first (so `Authorization=Bearer x=y` → `Bearer x=y`). /// keeps any `=` after the first (so `Authorization=Bearer x=y` → `Bearer x=y`).
fn parse_kv(s: &str) -> Vec<(String, String)> { fn parse_kv(s: &str) -> Vec<(String, String)> {

View file

@ -273,16 +273,13 @@ in
RuntimeDirectoryPreserve = "yes"; RuntimeDirectoryPreserve = "yes";
StateDirectory = "hyperhive"; StateDirectory = "hyperhive";
StateDirectoryMode = "0750"; StateDirectoryMode = "0750";
# OTEL auth-header secret, loaded onto hive-c0re's own unit so its # No OTEL credential here. hive-c0re's container-resource exporter
# per-agent container-resource metrics exporter can read it # targets this hive's own collector, which takes unauthenticated OTLP
# at $CREDENTIALS_DIRECTORY/otel-headers — via systemd, not a world # on the bridge; the upstream header belongs to the swarm tier
# path. Same secret the agent containers get (forwarded there via # (`swarm-otel.nix`), the only hop that leaves the swarm. Handing it to
# nspawn --load-credential); this just also hands it to c0re itself. # the daemon as well would put a secret on a process that has nowhere
# Empty list (no credential) when otel is off or no header is set. # to present it.
LoadCredential = LoadCredential =
lib.optional (
config.services.hyperhive.otel.enable && config.services.hyperhive.otel.headersCredential != null
) "otel-headers:${config.services.hyperhive.otel.headersCredential}"
# The swarm-queue client secret this hive authenticates with to # The swarm-queue client secret this hive authenticates with to
# publish its own status. `LoadCredential` and not a copy: root # publish its own status. `LoadCredential` and not a copy: root
# reads the plaintext at unit start and hive-core sees it 0400 # reads the plaintext at unit start and hive-core sees it 0400
@ -290,7 +287,7 @@ in
# and the daemon never needs read access to wherever it lives. # and the daemon never needs read access to wherever it lives.
# (The callout responder copies instead only because it # (The callout responder copies instead only because it
# delivers into a container, across a filesystem boundary.) # delivers into a container, across a filesystem boundary.)
++ lib.optional ( lib.optional (
config.services.hyperhive.swarm.statusPublish.clientSecretFile != null config.services.hyperhive.swarm.statusPublish.clientSecretFile != null
) "swarm-status-client.secret:${config.services.hyperhive.swarm.statusPublish.clientSecretFile}"; ) "swarm-status-client.secret:${config.services.hyperhive.swarm.statusPublish.clientSecretFile}";
# Sandboxing. hive-c0re is unprivileged (runs as hive-core, never # Sandboxing. hive-c0re is unprivileged (runs as hive-core, never