vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

Compare commits

base: vinzenz:865a1cc9acf84402216dc1d049904293a22b20c5
Branches Tags
vinzenz:main
..
compare: vinzenz:3b9e54d02870641bf454ca75aeda3df5f7fb62d8
Branches Tags
vinzenz:main
3 changed files with 41 additions and 247 deletions
Download patch file Download diff file Expand all files Collapse all files

105
docs/observability.md
View file

@ -16,41 +16,12 @@ services.hyperhive.otel = {
};
```
`enable` is the single gate. `endpoint` (required when enabled) is where
telemetry ultimately goes.
`enable` is the single gate. `endpoint` (required when enabled) is the OTLP
HTTP endpoint; hive-c0re injects it as `OTEL_EXPORTER_OTLP_ENDPOINT` into
every agent's systemd service via the generated meta flake.
**There is exactly one way telemetry leaves a hive: through the collector that
`enable` starts on the host.** Agents never talk to `endpoint` themselves —
they export unauthenticated to a bridge address only their own containers can
reach, and the collector forwards upstream with the auth header. So the
upstream credential exists in one place, on the host, and no agent ever holds
a copy.
⚠️ **The collector is therefore in the path of all telemetry.** It runs on the
same host as the agents and restarts on failure, and telemetry is not the
control plane — degraded telemetry is not degraded operation — but the export
no longer survives independently of anything host-side.
### what the agent→collector hop is and isn't
**It has no application-level auth.** The receiver takes any OTLP that reaches
it; what bounds who can reach it is the firewall — `exposeHostPorts` opens the
port on the bridge interface only. So "unauthenticated to a bridge address"
means *reachable from an agent container*, not *presents a credential*.
The consequence, stated because it is a choice rather than an oversight: **any
agent can push arbitrary OTLP, and the collector forwards it upstream under the
operator's credential.** It cannot tell a container's genuine Claude Code stats
from anything else shaped like OTLP arriving on that port — including data
smuggled out in resource attributes on an otherwise-legitimate export.
That is a **different risk from the one the collector fixes**, and strictly
smaller than what preceded it: before, every agent held the upstream credential
itself, so it could do all of the above *and* use the token anywhere else. The
collector removes the token and keeps the pipe. Agents are inside the trust
boundary (`docs/security.md`: capability = accepted risk), so an agent being
able to *send* is an accepted extension of that boundary — but it is not
closed by this design, and nothing here should be read as closing it.
Each agent's harness (hive-ag3nt) exports directly to the collector — the
pipeline keeps working even when hive-c0re is down.
## Options reference
@ -72,17 +43,17 @@ OTLP wire protocol, passed as `OTEL_EXPORTER_OTLP_PROTOCOL`. Accepted values:
### `services.hyperhive.otel.headersCredential` — string or null, default `null`
Absolute path to a secret file on the host holding the upstream auth header as
`NAME=value` (e.g. `Authorization=Bearer <token>`).
Absolute path to a secret file on the host whose contents become
`OTEL_EXPORTER_OTLP_HEADERS` (e.g. `Authorization=Bearer <token>`).
**Only the host collector reads it.** It arrives as an `EnvironmentFile` on the
collector's unit, so the value is never read by nix, never copied into the
store or the generated config, never passed in argv — and **never forwarded
into an agent container**. An agent cannot read the hive's upstream credential
because it is never given one.
hive-c0re forwards this host file into each agent container via
`systemd-nspawn --load-credential=otel-headers:<path>`; the inner harness unit
inherits it by name. The token is never copied into the nix store, generated
config, a bind mount, or argv.
Leave `null` if the upstream needs no auth header; the collector then sends
none rather than an empty one.
Leave `null` if the endpoint needs no auth header. A configured-but-missing
file is skipped with a log warning — OTEL still exports, just without the auth
header.
```nix
services.hyperhive.otel = {
@ -122,50 +93,6 @@ metrics on process exit, so interval tuning is not required for metrics to be
exported. A lower value gives more frequent intermediate flushes within
long-running turns — cosmetic, not a correctness knob.
## The host collector
`enable` starts an OpenTelemetry collector on the host. It is not optional and
there is no second path — that is the whole point:
```nix
services.hyperhive.otel = {
enable = true;
endpoint = "https://collector.example.com/otel"; # the upstream
headersCredential = "/run/secrets/otel-headers"; # only the host reads it
};
```
**Why it isn't a knob.** Exporting straight to `endpoint` means every agent
needs the credential to authenticate — and the harness delivers that token into
the agent's own `~/.claude/settings.json`, a file the agent can read. `0600`
protects it from other containers, not from the agent itself. As long as the
direct path stays *selectable*, that hole stays selectable; an option that can
reintroduce it is a hole with extra steps.
**`endpoint` keeps meaning "where telemetry ultimately goes."** The collector
does not redefine it — the agent-facing value is *derived*
(`http://<bridgeIp>:<collector.port>`), so an existing deployment's `endpoint`
keeps working unchanged. The bridge port is contributed to `exposeHostPorts`
automatically; there is nothing to open by hand.
### `services.hyperhive.otel.collector.port` — port, default `4318`
The OTLP/HTTP port the collector listens on, bound to the bridge IP only.
### `services.hyperhive.otel.collector.upstreamHeaderName` — string, default `"Authorization"`
Name of the header the collector sends upstream. The **value** comes from the
credential file at runtime (`EnvironmentFile` → `${env:<name>}`), never from
nix — so header names are config and header values are secrets, which is the
only split the collector's static header map can express.
⚠️ **`endpoint` must be valid for `protocol`.** The upstream exporter follows
`otel.protocol` (`grpc` → the gRPC exporter, otherwise OTLP/HTTP), and the gRPC
exporter takes an *address*: `https://host/path` is a legal
`OTEL_EXPORTER_OTLP_ENDPOINT` for HTTP but fails as gRPC with *"missing port in
address"*. The collector's config is validated at build time, so a mismatch is
a build error naming the reason rather than telemetry silently going nowhere.
## Network access
Agent containers can only reach the host on ports 80 and 443 by default. If
@ -185,10 +112,6 @@ services.hyperhive.otel.endpoint = "http://10.42.0.1:4318";
The bridge IP is the host's address on the `hvbr0` bridge, typically
`10.42.0.1`. See `docs/network.md::Reaching host services` for details.
⚠️ **You do not need either line for hyperhive's own telemetry** — `otel.enable`
contributes the collector's port and derives the endpoint itself. The above is
for pointing something *else* at a host-local service.
## Built-in resource labels
Every agent's export includes these resource attributes automatically:

24
nix/host-modules/hive-c0re/environment.nix
View file

@ -83,29 +83,15 @@ in
otel = config.services.hyperhive.otel;
in
{
# `otel.endpoint` means "where telemetry ultimately goes" and keeps
# that meaning; what agents are handed is the *first hop*, which is
# always this hive's own collector. Deriving it rather than
# redefining `endpoint` is what lets every existing deployment keep
# its configured value untouched.
HYPERHIVE_OTEL_ENDPOINT = "http://${config.services.hyperhive.network.bridgeIp}:${toString otel.collector.port}";
# The first hop is the collector's OTLP/HTTP receiver, which speaks
# protobuf regardless of what the upstream wants — `otel.protocol`
# describes the *upstream* link, and the collector's own exporter is
# what has to honour it (see nix/host-modules/otel.nix).
HYPERHIVE_OTEL_PROTOCOL = "http/protobuf";
HYPERHIVE_OTEL_ENDPOINT = otel.endpoint;
HYPERHIVE_OTEL_PROTOCOL = otel.protocol;
}
// lib.optionalAttrs (otel.extraResourceAttributes != "") {
HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES = otel.extraResourceAttributes;
}
# HYPERHIVE_OTEL_HEADERS_CREDENTIAL is deliberately NOT emitted, and
# its absence is the security half of this design. It is the variable
# that put the upstream token in an agent's own settings.json:
# host_config.rs forwards it into the container as an nspawn
# credential, and claude-settings.nix's `hive-otel-header` oneshot
# then writes the value into a file the agent can read. The collector
# holding the credential achieves nothing while the harness keeps
# handing out a copy — so there is exactly one holder, on the host.
// lib.optionalAttrs (otel.headersCredential != null) {
HYPERHIVE_OTEL_HEADERS_CREDENTIAL = otel.headersCredential;
}
// lib.optionalAttrs (otel.metricIntervalMs != null) {
HYPERHIVE_OTEL_METRIC_INTERVAL_MS = toString otel.metricIntervalMs;
}

157
nix/host-modules/otel.nix
View file

@ -16,19 +16,9 @@
enable = lib.mkEnableOption ''
hive-wide export of every agent's Claude Code stats (token usage,
cost, tool calls) to an OTLP endpoint via Claude Code's built-in
OpenTelemetry. One switch for all agents.
Enabling this also runs a collector on the host: there is exactly
one way telemetry leaves this hive, and it is through that
collector. Agents export unauthenticated to a bridge address only
their own containers can reach, and the collector is the only
holder of the upstream credential — an agent never sees it.
⚠️ The collector is therefore in the path of all telemetry. It runs
on the same host as the agents and restarts on failure, and
telemetry is not the control plane, so degraded telemetry is not
degraded operation — but the export no longer survives independently
of anything host-side
OpenTelemetry. One switch for all agents; each harness exports
directly to the collector, so it keeps working even when hive-c0re
is down
'';
endpoint = lib.mkOption {
@ -62,19 +52,16 @@
default = null;
example = "/run/secrets/otel-headers";
description = ''
Absolute path to an operator-provided secret file holding the
upstream auth header as `NAME=value` (e.g.
`Authorization=Bearer <token>`).
**Only the host-side collector reads this.** It reaches the
collector as an `EnvironmentFile`, so the value is never read by
nix, never copied into the store or the generated config, and
never passed in argv — and it is never forwarded into an agent
container, which is the point of the collector existing. Must be
absolute.
Leave null if the upstream needs no auth header; the collector
then sends none rather than an empty one.
Absolute path to an operator-provided secret file whose contents
become `OTEL_EXPORTER_OTLP_HEADERS` (e.g.
`Authorization=Bearer <token>`). hive-c0re forwards this host
file into each agent container's credential store via
systemd-nspawn `--load-credential=otel-headers:<path>`; the inner
harness unit inherits it by name (`LoadCredential`), so the token
is never copied into the nix store, the generated config, a bind
mount, or argv. Must be absolute. Leave null if the endpoint
needs no auth header. A configured-but-missing file is skipped
with a log warning (OTEL still exports, without the auth header).
'';
};
@ -101,41 +88,6 @@
'';
};
collector.upstreamHeaderName = lib.mkOption {
type = lib.types.str;
default = "Authorization";
description = ''
Name of the HTTP header the collector sends upstream, whose
*value* comes from `headersCredential`.
The name is here and the value is not, and that split is forced
rather than chosen: the collector models exporter headers as a
static map, so rendering them means nix reading the value — the
one thing `headersCredential` being a path exists to prevent.
A name is public, a value is not.
⇒ exactly one header is expressible this way. A credential
carrying several (`a=1,b=2`) would be read as a single value,
which is why the shape is a named header rather than an opaque
blob: a second header has to be *declared*, not smuggled.
'';
};
collector.port = lib.mkOption {
type = lib.types.port;
default = 4318;
description = ''
Port the collector's OTLP/HTTP receiver listens on, at
`services.hyperhive.network.bridgeIp`. 4318 is the OTLP/HTTP
default.
The port is contributed to
`services.hyperhive.network.exposeHostPorts`, which opens it on
the bridge interface only — so it is reachable from agent
containers and not from the outside world.
'';
};
metricIntervalMs = lib.mkOption {
type = lib.types.nullOr lib.types.ints.positive;
default = null;
@ -154,79 +106,12 @@
};
};
config = lib.mkMerge [
(lib.mkIf config.services.hyperhive.c0re.enable {
assertions = lib.optionals config.services.hyperhive.otel.enable [
{
assertion = config.services.hyperhive.otel.endpoint != "";
message = "services.hyperhive.otel.enable is true but services.hyperhive.otel.endpoint is empty.";
}
];
})
(lib.mkIf config.services.hyperhive.otel.enable (
(
let
otel = config.services.hyperhive.otel;
listen = "${config.services.hyperhive.network.bridgeIp}:${toString otel.collector.port}";
# `otel.protocol` describes the UPSTREAM link and always did.
# Inserting a collector splits the path in two, and the
# upstream half is the one that has to keep honouring it — so
# the exporter is chosen by it, rather than the option quietly
# becoming "how agents talk to the collector". The agent half
# is pinned to OTLP/HTTP by the receiver below (derived in
# hive-c0re/environment.nix).
grpcUpstream = otel.protocol == "grpc";
upstreamName = if grpcUpstream then "otlp" else "otlphttp";
upstream = {
endpoint = otel.endpoint;
}
// lib.optionalAttrs (otel.headersCredential != null) {
# The value is interpolated by the collector at runtime from
# its environment, never by nix. `EnvironmentFile` below is
# what puts it there. No credential configured means no
# header at all — an upstream that needs no auth is a
# legitimate deployment, and rendering `${env:…}` for a
# variable nothing sets would send the literal.
headers.${otel.collector.upstreamHeaderName} = "\${env:${otel.collector.upstreamHeaderName}}";
}
// lib.optionalAttrs (otel.protocol == "http/json") { encoding = "json"; };
in
{
# Reachable from agent containers and nowhere else: this opens
# the port on the bridge interface only.
services.hyperhive.network.exposeHostPorts = [ otel.collector.port ];
services.opentelemetry-collector = {
enable = true;
# `validateConfigFile` defaults to `isStorePath configFile`,
# and `configFile` is null on the `settings` path — so the
# upstream default is OFF for exactly the way this module
# configures it. Turning it on runs `otelcol validate` at
# build time, which is the collector checking its own config.
# ⚠️ It parses; it does not prove a sample arrives.
validateConfigFile = true;
settings = {
receivers.otlp.protocols.http.endpoint = listen;
exporters.${upstreamName} = upstream;
service.pipelines.metrics = {
receivers = [ "otlp" ];
exporters = [ upstreamName ];
};
};
};
# The credential file is already `NAME=value`, which is
# systemd's EnvironmentFile format — so the secret reaches the
# process as an environment variable without ever being read by
# nix, written to the store, or passed in argv.
systemd.services.opentelemetry-collector.serviceConfig =
lib.optionalAttrs (otel.headersCredential != null)
{
EnvironmentFile = otel.headersCredential;
};
}
)
))
];
config = lib.mkIf config.services.hyperhive.c0re.enable {
assertions = lib.optionals config.services.hyperhive.otel.enable [
{
assertion = config.services.hyperhive.otel.endpoint != "";
message = "services.hyperhive.otel.enable is true but services.hyperhive.otel.endpoint is empty.";
}
];
};
}