vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

feat(#3125): the hive tier forwards to the swarm tier and holds nothing

Browse source 
The hive collector's only exporter becomes the swarm's collector, and
the upstream credential, the metrics-store exporter and the choice of
destination all move one tier up.

Its assertion goes with them: 'endpoint or a local store' was the right
rule while this tier picked the destination, and is the wrong one now.
A hive that runs no swarm services has neither, forwards to a swarm
collector elsewhere, and is correctly configured — the rule that
replaces it lives in swarm-otel.nix, where the destinations are.

The option descriptions here described a topology with one collector in
it: endpoint and protocol are not what agents are handed (they get the
derived first hop, see hive-c0re/environment.nix), and the credential is
not read by this tier.
This commit is contained in:
atlas 2026-08-18 13:09:49 +02:00 • committed by mara
commit 094a54e785
1 changed files with 105 additions and 151 deletions
Download patch file Download diff file Expand all files Collapse all files

250
nix/host-modules/otel.nix
View file

@ -1,11 +1,18 @@
# Hive-wide OTEL stats export. Set ONCE here at host level; the # Hive-wide OTEL stats export, and the HIVE tier of the collector pair.
# meta-flake renderer (`hive-c0re/src/meta.rs::otel_config`) reads the # Set ONCE here at host level; the meta-flake renderer
# HYPERHIVE_OTEL_* env exported off hive-c0re's unit (see # (`hive-c0re/src/meta.rs::otel_config`) reads the HYPERHIVE_OTEL_* env
# ./hive-c0re) and injects the matching `hyperhive.otel.*` build-time # exported off hive-c0re's unit (see ./hive-c0re) and injects the matching
# config into EVERY agent (mirroring the CA-cert injection), so each # `hyperhive.otel.*` build-time config into EVERY agent (mirroring the
# agent's harness exports its own Claude Code stats directly to the # CA-cert injection), so each agent's harness exports its own Claude Code
# collector. There is no per-agent opt-in — this is the single switch # stats directly to the collector. There is no per-agent opt-in — this is
# for the whole hive. # the single switch for the whole hive.
#
# This tier receives from this hive's agents and forwards to the swarm's
# collector (./swarm-otel.nix). It holds no credential and picks no
# destination: an agent's samples cross a hive boundary exactly once, and
# what happens after that is the swarm's decision, not a hive's. The
# upstream options declared below describe that far end and are read one
# tier up — they stay here because they mean what they have always meant.
{ {
lib, lib,
config, config,
@ -18,11 +25,15 @@
cost, tool calls) to an OTLP endpoint via Claude Code's built-in cost, tool calls) to an OTLP endpoint via Claude Code's built-in
OpenTelemetry. One switch for all agents. OpenTelemetry. One switch for all agents.
Enabling this also runs a collector on the host: there is exactly Enabling this also runs this hive's collector on the host: there is
one way telemetry leaves this hive, and it is through that exactly one way telemetry leaves this hive, and it is through that
collector. Agents export unauthenticated to a bridge address only collector. Agents export unauthenticated to a bridge address only
their own containers can reach, and the collector is the only their own containers can reach.
holder of the upstream credential — an agent never sees it.
That collector forwards to the swarm's
({option}`services.hyperhive.swarm.otel.enable`), which holds the
upstream credential and writes the swarm's store. So an agent never
sees the credential, and neither does this tier.
⚠️ The collector is therefore in the path of all telemetry. It runs ⚠️ The collector is therefore in the path of all telemetry. It runs
on the same host as the agents and restarts on failure, and on the same host as the agents and restarts on failure, and
@ -36,15 +47,19 @@
default = ""; default = "";
example = "https://collector.example.com/otel"; example = "https://collector.example.com/otel";
description = '' description = ''
Upstream OTLP endpoint, set as `OTEL_EXPORTER_OTLP_ENDPOINT` for Upstream OTLP endpoint: where telemetry ultimately goes, after it
every agent. has left the swarm.
Required when `enable` is true, **unless** this host runs the Read by the swarm's collector
swarm's metrics store ({option}`services.hyperhive.swarm.otel.enable`), which is the only
({option}`services.hyperhive.swarm.victoriametrics.enable`) — that tier that holds the upstream credential. An agent is handed the
store is a destination in its own right, and with both configured *first* hop instead — this hive's own collector — so this value is
telemetry goes to both. With neither, `enable` is refused rather never given to a container.
than silently exporting nowhere.
Optional. Leave it empty and the swarm's own metrics store
({option}`services.hyperhive.swarm.victoriametrics.enable`) is the
destination; that is a complete deployment, not a degraded one.
Set both and telemetry goes to both.
''; '';
}; };
@ -56,7 +71,12 @@
]; ];
default = "http/protobuf"; default = "http/protobuf";
description = '' description = ''
OTLP wire protocol, set as `OTEL_EXPORTER_OTLP_PROTOCOL`. OTLP wire protocol for the **upstream** link, honoured by the
swarm collector's exporter.
Not what agents speak: their first hop is this hive's collector,
whose OTLP/HTTP receiver takes protobuf whatever the upstream
wants (see `hive-c0re/environment.nix`).
''; '';
}; };
@ -73,12 +93,12 @@
upstream auth header as `NAME=value` (e.g. upstream auth header as `NAME=value` (e.g.
`Authorization=Bearer <token>`). `Authorization=Bearer <token>`).
**Only the host-side collector reads this.** It reaches the **Only the swarm's collector reads this** — the one tier that
collector as an `EnvironmentFile`, so the value is never read by talks to the upstream. It arrives as an `EnvironmentFile`, so the
nix, never copied into the store or the generated config, and value is never read by nix, never copied into the store or the
never passed in argv — and it is never forwarded into an agent generated config, and never passed in argv; and it reaches
container, which is the point of the collector existing. Must be neither an agent container nor this hive's own collector, which
absolute. is the point of the tiers existing. Must be absolute.
Leave null if the upstream needs no auth header; the collector Leave null if the upstream needs no auth header; the collector
then sends none rather than an empty one. then sends none rather than an empty one.
@ -161,131 +181,65 @@
}; };
}; };
config = lib.mkMerge [ config = lib.mkIf config.services.hyperhive.otel.enable (
(lib.mkIf config.services.hyperhive.c0re.enable { let
assertions = lib.optionals config.services.hyperhive.otel.enable [ otel = config.services.hyperhive.otel;
{ listen = "${config.services.hyperhive.network.bridgeIp}:${toString otel.collector.port}";
# Telemetry has to go SOMEWHERE, but "somewhere" stopped meaning swarmName = "otlphttp/swarm";
# "an upstream endpoint" once the swarm grew its own store: a hive in
# running `swarm.victoriametrics` is a complete destination on its {
# own, and requiring an external endpoint as well would make the # Reachable from agent containers and nowhere else: this opens
# all-local mode impossible to express. # the port on the bridge interface only.
# services.hyperhive.network.exposeHostPorts = [ otel.collector.port ];
# This only ever relaxes the old rule — every config that passed
# before still passes.
assertion =
config.services.hyperhive.otel.endpoint != ""
|| config.services.hyperhive.swarm.victoriametrics.enable;
message = ''
services.hyperhive.otel.enable is true but telemetry has nowhere
to go: services.hyperhive.otel.endpoint is empty and
services.hyperhive.swarm.victoriametrics.enable is false.
Set the endpoint to export upstream, or enable the swarm's services.opentelemetry-collector = {
metrics store to keep telemetry on this host. enable = true;
''; # `validateConfigFile` defaults to `isStorePath configFile`,
} # and `configFile` is null on the `settings` path — so the
]; # upstream default is OFF for exactly the way this module
}) # configures it. Turning it on runs `otelcol validate` at
# build time, which is the collector checking its own config.
# ⚠️ It is a PARSER, not a wiring check, and the gap is wider
# than "no sample was sent": measured 2026-08-15, `validate`
# ACCEPTS a receiver naming an auth extension that is absent
# from the build, and the collector then dies at startup with
# `Failed to start component`. So a green build does not
# prove this config STARTS, never mind that a sample arrives.
validateConfigFile = true;
settings = {
receivers.otlp.protocols.http.endpoint = listen;
(lib.mkIf config.services.hyperhive.otel.enable ( # One destination, and it is the swarm's collector. This tier
( # holds no upstream credential and writes no store: it receives
let # from this hive's agents and forwards, which is the whole of
otel = config.services.hyperhive.otel; # its job. Everything that decides where telemetry ultimately
listen = "${config.services.hyperhive.network.bridgeIp}:${toString otel.collector.port}"; # goes lives one tier up, in ./swarm-otel.nix.
# `otel.protocol` describes the UPSTREAM link and always did. exporters.${swarmName} = {
# Inserting a collector splits the path in two, and the # ⚠️ Plain `endpoint`, and the exporter beside this one in
# upstream half is the one that has to keep honouring it — so # ./swarm-otel.nix warns against exactly that spelling — read
# the exporter is chosen by it, rather than the option quietly # both before "fixing" either. The difference is the far end,
# becoming "how agents talk to the collector". The agent half # not the exporter: `endpoint` is a BASE that otlphttp appends
# is pinned to OTLP/HTTP by the receiver below (derived in # `/v1/metrics` to, which is precisely the path an OTLP/HTTP
# hive-c0re/environment.nix). # receiver serves. VictoriaMetrics is the odd one out, serving
vmCfg = config.services.hyperhive.swarm.victoriametrics; # OTLP at `/opentelemetry/api/v1/push`, and that is why the
# Two independent destinations, either of which may be absent: an # store exporter needs `metrics_endpoint` while this one must
# upstream the operator named, and the swarm's own store when this # not have it.
# host runs it. The assertion above guarantees at least one.
upstreamConfigured = otel.endpoint != "";
localStore = vmCfg.enable;
storeName = "otlphttp/victoriametrics";
store = {
# ⚠️ `metrics_endpoint`, NOT `endpoint`, and the difference is
# invisible until you read the far end: `endpoint` is a BASE that
# otlphttp appends `/v1/metrics` to, while VictoriaMetrics serves
# OTLP at `/opentelemetry/api/v1/push`. With `endpoint` the
# collector still answers 200 to its own clients and the samples
# are silently posted to a path that does not exist.
# `metrics_endpoint` is used verbatim.
# #
# Measured end-to-end rather than read: a real sample crossed a # Addressed by the option rather than by a loopback literal:
# real collector into a real store, and the same probe with # the default already points at the co-located tier, and a
# `endpoint` never arrived — see `state/probe-3265-collector-to-vm.sh`. # hive whose swarm collector lives elsewhere then names it in
metrics_endpoint = "http://127.0.0.1:${toString vmCfg.port}/opentelemetry/api/v1/push"; # config instead of needing this file changed. A loopback
# literal is correct only while listener and caller share a
# netns — the assumption that cost #2860 and #3363.
endpoint = config.services.hyperhive.swarm.otel.url;
}; };
grpcUpstream = otel.protocol == "grpc"; service.pipelines.metrics = {
upstreamName = if grpcUpstream then "otlp" else "otlphttp"; receivers = [ "otlp" ];
upstream = { exporters = [ swarmName ];
endpoint = otel.endpoint;
}
// lib.optionalAttrs (otel.headersCredential != null) {
# The value is interpolated by the collector at runtime from
# its environment, never by nix. `EnvironmentFile` below is
# what puts it there. No credential configured means no
# header at all — an upstream that needs no auth is a
# legitimate deployment, and rendering `${env:…}` for a
# variable nothing sets would send the literal.
headers.${otel.collector.upstreamHeaderName} = "\${env:${otel.collector.upstreamHeaderName}}";
}
// lib.optionalAttrs (otel.protocol == "http/json") { encoding = "json"; };
in
{
# Reachable from agent containers and nowhere else: this opens
# the port on the bridge interface only.
services.hyperhive.network.exposeHostPorts = [ otel.collector.port ];
services.opentelemetry-collector = {
enable = true;
# `validateConfigFile` defaults to `isStorePath configFile`,
# and `configFile` is null on the `settings` path — so the
# upstream default is OFF for exactly the way this module
# configures it. Turning it on runs `otelcol validate` at
# build time, which is the collector checking its own config.
# ⚠️ It is a PARSER, not a wiring check, and the gap is wider
# than "no sample was sent": measured 2026-08-15, `validate`
# ACCEPTS a receiver naming an auth extension that is absent
# from the build, and the collector then dies at startup with
# `Failed to start component`. So a green build does not
# prove this config STARTS, never mind that a sample arrives.
validateConfigFile = true;
settings = {
receivers.otlp.protocols.http.endpoint = listen;
exporters =
lib.optionalAttrs upstreamConfigured { ${upstreamName} = upstream; }
// lib.optionalAttrs localStore { ${storeName} = store; };
service.pipelines.metrics = {
receivers = [ "otlp" ];
# Fan-out, not a choice: with both configured the same
# samples go upstream AND into the swarm's store. A local
# store is for looking at this swarm; an upstream is for
# whoever aggregates across swarms, and neither replaces
# the other.
exporters = lib.optional upstreamConfigured upstreamName ++ lib.optional localStore storeName;
};
};
}; };
};
# The credential file is already `NAME=value`, which is };
# systemd's EnvironmentFile format — so the secret reaches the }
# process as an environment variable without ever being read by );
# nix, written to the store, or passed in argv.
systemd.services.opentelemetry-collector.serviceConfig =
lib.optionalAttrs (otel.headersCredential != null)
{
EnvironmentFile = otel.headersCredential;
};
}
)
))
];
} }