otel: scrape each collector's own loss counters

A collector's `refused` / `failed` / queue-depth counters are the only
signal that says telemetry is being dropped, and nothing read them at
any tier — so a collector losing records looked exactly like a quiet
system.

The hive tier could not be scraped without first naming its port. 8888
is the collector's built-in default and appeared in no config, which is
also why nothing comparing configured ports could see it clash with a
co-located collector — swarm-otel.nix sidesteps 8888 by hand for that
reason, and says so. Declaring the port and binding it explicitly makes
the value comparable; wiring the scrape is then one entry per tier.

Extending the port-collision assertion to cover it is deliberately left
out: that belongs with the other port work, and coupling a collision
fix to a scraping fix makes both harder to review.

Gate: 101 module properties hold, was 95. The six cases pin the rendered
scrape job rather than the option; the metrics pipeline naming the
prometheus receiver, a path never emitted on any hive before this since
the hive tier's scrapeTargets was empty everywhere; the `readers`
spelling, with a control so a missing telemetry block cannot pass the
port check vacuously; the swarm tier's own entry; the two tiers not
claiming the same port; and the absence arm, a hive with no collector
declaring no target.

scrapeTargets' description said "Empty by default, and that is the
shipped case". This makes that false, so the paragraph moves with it.
This commit is contained in:
atlas 2026-09-12 09:18:30 +02:00 committed by mara
commit 98945d4c5a
3 changed files with 130 additions and 4 deletions

View file

@ -447,6 +447,16 @@ let
otel.clientSecretFile = "/var/lib/hive-otel-oidc/client.secret";
};
hiveOtelPipelines = hiveOtel.services.opentelemetry-collector.settings.service.pipelines;
hiveOtelSettings = hiveOtel.services.opentelemetry-collector.settings;
# The hive tier's rendered scrape list. Same reasoning as `scrapeJob` for
# the swarm tier — the option is one string, what prometheus is handed is a
# job — but this collector is a host service, so the path to it differs.
hiveScrapeJob =
job:
lib.findFirst (
c: c.job_name == job
) null hiveOtelSettings.receivers.prometheus.config.scrape_configs;
# Two hives in the roster, which no other fixture here has: every one of
# them declares `swarm.hives.h1` alone, so a per-hive arm written against
@ -1368,6 +1378,70 @@ let
!(builtins.elem "deltatocumulative" hiveOtelPipelines.logs.processors)
&& builtins.elem "deltatocumulative" hiveOtelPipelines.metrics.processors;
}
{
# The counters that say telemetry is being LOST — refused, failed,
# queue depth — are served on loopback and reach no store unless
# something reads them. Read off the rendered job rather than the
# option: only the job is what prometheus actually requests.
name = "the hive collector scrapes its own telemetry endpoint";
ok =
let
j = hiveScrapeJob "collector";
in
j != null && j.static_configs == [ { targets = [ "127.0.0.1:8888" ]; } ];
}
{
# A `prometheus` receiver no pipeline names collects nothing while
# rendering and starting perfectly, so the scrape above is inert
# without this. The path is newly reachable: until the hive tier had a
# target of its own, this receiver was never emitted on any hive.
name = "the hive metrics pipeline names the prometheus receiver the self-scrape needs";
ok = builtins.elem "prometheus" hiveOtelPipelines.metrics.receivers;
}
{
# `metrics.address` is the spelling that looks right and is rejected by
# this collector version. The first clause is the control: without it a
# missing telemetry block would pass the port check vacuously.
name = "the hive collector binds its telemetry port through readers, not address";
ok =
let
m = hiveOtelSettings.service.telemetry.metrics;
in
!(m ? address) && (lib.head m.readers).pull.exporter.prometheus.port == 8888;
}
{
# Same gap one tier up, and it needs its own arm: this collector
# already had six scrape targets, so a pass here is about the seventh
# rather than about the receiver existing at all.
name = "the swarm collector scrapes its own telemetry endpoint";
ok =
let
j = scrapeJob baoWithCollector "collector";
in
j != null && j.static_configs == [ { targets = [ "127.0.0.1:8889" ]; } ];
}
{
# Both collectors share a network namespace whenever they are
# co-located, and this port appears in no config the port-collision
# assertion can read — so equal defaults mean the second to start dies
# at `bind()`. Pinned as a case rather than an assertion: enforcing it
# belongs with the other port checks, not here.
name = "the two collector tiers do not claim the same self-telemetry port";
ok =
let
portOf = s: (lib.head s.service.telemetry.metrics.readers).pull.exporter.prometheus.port;
in
portOf hiveOtelSettings != portOf (otelSettings baoWithCollector);
}
{
# The absence arm for the case above, and the option's own rule — a
# service declares its entry under its own `enable` — made checkable.
# Without it, moving the assignment outside the collector's `mkIf`
# passes every arm above while handing a collector-less hive a scrape
# target for a port nothing binds.
name = "a hive with no collector declares no self-scrape target";
ok = bare.services.hyperhive.otel.scrapeTargets == { };
}
{
# Read against the roster the fixture declares rather than against
# names spelled here: an arm naming `h1` passes on a single-hive