otel: scrape each collector's own loss counters

A collector's `refused` / `failed` / queue-depth counters are the only
signal that says telemetry is being dropped, and nothing read them at
any tier — so a collector losing records looked exactly like a quiet
system.

The hive tier could not be scraped without first naming its port. 8888
is the collector's built-in default and appeared in no config, which is
also why nothing comparing configured ports could see it clash with a
co-located collector — swarm-otel.nix sidesteps 8888 by hand for that
reason, and says so. Declaring the port and binding it explicitly makes
the value comparable; wiring the scrape is then one entry per tier.

Extending the port-collision assertion to cover it is deliberately left
out: that belongs with the other port work, and coupling a collision
fix to a scraping fix makes both harder to review.

Gate: 101 module properties hold, was 95. The six cases pin the rendered
scrape job rather than the option; the metrics pipeline naming the
prometheus receiver, a path never emitted on any hive before this since
the hive tier's scrapeTargets was empty everywhere; the `readers`
spelling, with a control so a missing telemetry block cannot pass the
port check vacuously; the swarm tier's own entry; the two tiers not
claiming the same port; and the absence arm, a hive with no collector
declaring no target.

scrapeTargets' description said "Empty by default, and that is the
shipped case". This makes that false, so the paragraph moves with it.
This commit is contained in:
atlas 2026-09-12 09:18:30 +02:00 committed by mara
commit 98945d4c5a
3 changed files with 130 additions and 4 deletions

View file

@ -193,6 +193,24 @@ in
'';
};
telemetryPort = lib.mkOption {
type = lib.types.port;
default = 8888;
description = ''
Port this collector serves its **own** metrics on queue depth,
refused and dropped samples, exporter failures. How you find out
that telemetry is being lost.
8888 is the collector's built-in default; declaring it here makes
the value *configured* rather than implied, which is the point of
the option. An undeclared port is invisible to anything comparing
ports, so a second collector on the same host picking the same
number fails at `bind()` rather than at eval
`services.hyperhive.swarm.otel.telemetryPort` sidesteps 8888 by
hand for that reason.
'';
};
scrapeTargets = lib.mkOption {
type = lib.types.attrsOf lib.types.str;
default = { };
@ -221,10 +239,11 @@ in
onward, which is one hop for the whole hive rather than one per
target.
Empty by default, and that is the shipped case no scrape
receiver is emitted at all, because a `prometheus` receiver with
nothing to scrape is a config that renders, starts and collects
nothing.
No longer empty on a default hive: this collector's own endpoint
is always in here, so the scrape receiver is emitted everywhere.
The `!= { }` guards around it stay meaningful the set can still
be forced empty, and a `prometheus` receiver with nothing to
scrape is a config that renders, starts and collects nothing.
'';
};
@ -317,6 +336,15 @@ in
# the port on the bridge interface only.
services.hyperhive.network.exposeHostPorts = [ otel.collector.port ];
# The collector scrapes itself. Its own counters — refused, failed,
# queue depth — are the only signal that says telemetry is being
# dropped, and they reach no store unless something reads them.
#
# Loopback, so this deliberately does NOT go through
# `exposeHostPorts`: the endpoint is for the collector beside it,
# not for agent containers.
services.hyperhive.otel.scrapeTargets.collector = "127.0.0.1:${toString otel.telemetryPort}";
services.opentelemetry-collector = {
enable = true;
# Contrib, matching the swarm tier (./swarm-otel.nix). The upstream
@ -556,6 +584,23 @@ in
processors = [ "resourcedetection" ];
exporters = [ swarmName ];
};
# Bound explicitly rather than left to the collector's built-in
# default, so `telemetryPort` is a value something can compare
# instead of an assumption.
#
# ⚠️ `metrics.address` is the spelling that looks right and is
# rejected by this collector version — ./swarm-otel.nix documents
# the exact error it produces. `readers` is the schema it accepts,
# and the difference is a startup failure rather than a warning.
service.telemetry.metrics.readers = [
{
pull.exporter.prometheus = {
host = "127.0.0.1";
port = otel.telemetryPort;
};
}
];
}
// lib.optionalAttrs senderAuth {
extensions.${authName} = {