vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

otel: scrape each collector's own loss counters

Browse source 
A collector's `refused` / `failed` / queue-depth counters are the only
signal that says telemetry is being dropped, and nothing read them at
any tier — so a collector losing records looked exactly like a quiet
system.

The hive tier could not be scraped without first naming its port. 8888
is the collector's built-in default and appeared in no config, which is
also why nothing comparing configured ports could see it clash with a
co-located collector — swarm-otel.nix sidesteps 8888 by hand for that
reason, and says so. Declaring the port and binding it explicitly makes
the value comparable; wiring the scrape is then one entry per tier.

Extending the port-collision assertion to cover it is deliberately left
out: that belongs with the other port work, and coupling a collision
fix to a scraping fix makes both harder to review.

Gate: 101 module properties hold, was 95. The six cases pin the rendered
scrape job rather than the option; the metrics pipeline naming the
prometheus receiver, a path never emitted on any hive before this since
the hive tier's scrapeTargets was empty everywhere; the `readers`
spelling, with a control so a missing telemetry block cannot pass the
port check vacuously; the swarm tier's own entry; the two tiers not
claiming the same port; and the absence arm, a hive with no collector
declaring no target.

scrapeTargets' description said "Empty by default, and that is the
shipped case". This makes that false, so the paragraph moves with it.
This commit is contained in:
atlas 2026-09-12 09:18:30 +02:00 • committed by mara
commit 98945d4c5a
3 changed files with 130 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

7
nix/host-modules/swarm-otel.nix
View file

@ -533,6 +533,13 @@ in
# single hive's receiver refusing pushes, not the process dying.
services.hyperhive.swarm.otel.journaldUnits = [ "opentelemetry-collector" ];
# The metrics counterpart to the journal line above, closing the same
# gap from the other side: the journal says the process is alive, these
# counters say whether it is dropping what it receives. Loopback works
# here because `privateNetwork = false` — this container shares the
# host's netns, so `127.0.0.1` is where `telemetryPort` is bound.
services.hyperhive.swarm.otel.scrapeTargets.collector = "127.0.0.1:${toString cfg.telemetryPort}";
# OTLP/HTTP, not a browsable UI, but the same reverse-proxy shape as
# every sibling swarm service: TLS terminates here, then plain http to
# the co-located container over loopback (shared netns, like the store