vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(#3520): scrape swarm-service prometheus endpoints at the swarm tier

Browse source 
Nothing in this deployment read a Prometheus endpoint, so enabling
/metrics on a managed service added an endpoint and no data. The swarm
collector receives OTLP pushes and does not scrape; VictoriaMetrics
stores what is pushed and has no scrape config. The pipeline was entirely
push-based and every such service is pull-based.

Adds a prometheus receiver, a resource/swarm processor and a
metrics/swarm pipeline alongside the per-hive ones.

The pipeline is separate because that is the ruling, not for tidiness:
every resource/<hive> processor UPSERTS a hive key, so a scraped swarm
sample routed through one would acquire the single label a swarm-level
service must not have. Keeping it out makes the absence structural rather
than something to remember to strip, the same way hive stays a property
of which authenticated receiver accepted a push.

Targets come from an option each service fills in from its own module,
under its own enable, rather than a list assembled here. That is what
puts the scraper and the target on the same host by construction: an
entry exists only where the service that named it runs. Co-location is
true of the all-local deployment and is not a guarantee, and that is
exactly the case where assuming it is invisible.

All three additions MERGE with the per-hive attrsets rather than
replacing them. A plain assignment would drop every hive's receiver,
processor and pipeline and still render a config the collector starts
cleanly on.

Empty target set emits no receiver, no processor and no pipeline — an
enabled scraper with nothing to scrape is the inert configuration this
issue is about, and the target set ships empty here because the targets
themselves are separate issues.

Formatting verified with nix fmt. The evaluation gate is not written yet;
nothing here has been evaluated against a fixture.
This commit is contained in:
atlas 2026-08-19 19:58:46 +02:00 • committed by mara
commit ff5c76c42f
1 changed files with 114 additions and 41 deletions
Download patch file Download diff file Expand all files Collapse all files

155
nix/host-modules/swarm-otel.nix
View file

@ -181,6 +181,33 @@ in
addresses its siblings by name.
'';
};
scrapeTargets = lib.mkOption {
type = lib.types.attrsOf lib.types.str;
default = { };
example = lib.literalExpression ''{ forgejo = "127.0.0.1:3000"; }'';
description = ''
Prometheus exposition endpoints this collector scrapes, as
`<job name> = "<host>:<port>"`.
**A service declares its own entry, from its own module, under its
own `enable`.** That is what puts the scraper and the target on the
same host by construction rather than by luck: an entry exists only
where the service that named it runs. Do not assemble the list here.
Every swarm service being co-located is a property of the all-local
deployment, not a guarantee — and that is precisely the case where
the difference is invisible until a swarm splits across hosts.
Samples land in a swarm-level pipeline that stamps `swarm` and
**never** `hive`: a swarm service does not belong to a hive, and
`hive` stays a property of which authenticated receiver accepted a
push, not something a scrape can acquire.
Empty by default, in which case no scrape receiver, processor or
pipeline is emitted at all — an enabled scraper with nothing to
scrape is the inert configuration this option exists to avoid.
'';
};
};
config = lib.mkIf (config.services.hyperhive.enable && cfg.enable) {
@ -387,15 +414,32 @@ in
# an auth claim yields nothing — silently, with a healthy
# startup), and one receiver holding many credentials never
# reveals which one matched.
receivers = lib.mapAttrs' (
h: p:
lib.nameValuePair "otlp/${h}" {
protocols.http = {
endpoint = "127.0.0.1:${toString p}";
auth.authenticator = "oidc/${h}";
};
}
) hivePorts;
receivers =
lib.mapAttrs' (
h: p:
lib.nameValuePair "otlp/${h}" {
protocols.http = {
endpoint = "127.0.0.1:${toString p}";
auth.authenticator = "oidc/${h}";
};
}
) hivePorts
# MERGED with the per-hive receivers, never assigned over
# them. A plain assignment here would drop every hive's
# receiver and still render a valid config that starts
# cleanly — the collector has no opinion about how many
# pipelines it was supposed to have.
#
# Only emitted when a service has actually declared a
# target: a `prometheus` receiver with nothing to scrape is
# the shape this whole issue is about, a config that renders
# and deploys perfectly while adding no data.
// lib.optionalAttrs (cfg.scrapeTargets != { }) {
prometheus.config.scrape_configs = lib.mapAttrsToList (job: target: {
job_name = job;
static_configs = [ { targets = [ target ]; } ];
}) cfg.scrapeTargets;
};
exporters =
lib.optionalAttrs vmCfg.enable {
@ -452,14 +496,28 @@ in
# `exporterNames` is shared by every pipeline — where a
# sample goes is a property of this tier, not of the hive
# that sent it.
service.pipelines = lib.mapAttrs' (
h: _:
lib.nameValuePair "metrics/${h}" {
receivers = [ "otlp/${h}" ];
processors = [ "resource/${h}" ];
exporters = exporterNames;
}
) hivePorts;
service.pipelines =
lib.mapAttrs' (
h: _:
lib.nameValuePair "metrics/${h}" {
receivers = [ "otlp/${h}" ];
processors = [ "resource/${h}" ];
exporters = exporterNames;
}
) hivePorts
# Its OWN pipeline, and that separation is the ruling, not a
# tidiness choice: every `resource/<hive>` above UPSERTS a
# `hive` key, so a scraped swarm sample routed through any of
# them would acquire the one label a swarm-level service must
# not have. Keeping it out of them makes the absence
# structural rather than something to remember to strip.
// lib.optionalAttrs (cfg.scrapeTargets != { }) {
"metrics/swarm" = {
receivers = [ "prometheus" ];
processors = [ "resource/swarm" ];
exporters = exporterNames;
};
};
}
// {
extensions = lib.mapAttrs' (
@ -507,36 +565,51 @@ in
# processor is the whole attribution boundary — the value
# is a constant per receiver, so it says which hive
# authenticated, not which hive claimed to be sending.
processors = lib.mapAttrs' (
h: _:
lib.nameValuePair "resource/${h}" {
attributes = [
{
key = "hive";
value = h;
action = "upsert";
}
# Stamped here rather than on a separate upstream-only
# pipeline, which would double the pipeline count to
# withhold one constant label from the local store. It
# is redundant there — one VictoriaMetrics per swarm, so
# every series in it already belongs to this swarm — but
# a constant label multiplies no series, and it means
# what LEAVES and what STAYS have the same shape.
#
# Upstream is where it stops being redundant: that is the
# one hop where several swarms can land in one store, and
# samples that cannot name their swarm collide there
# exactly as hives collided here before per-hive
# receivers existed.
processors =
lib.mapAttrs' (
h: _:
lib.nameValuePair "resource/${h}" {
attributes = [
{
key = "hive";
value = h;
action = "upsert";
}
# Stamped here rather than on a separate upstream-only
# pipeline, which would double the pipeline count to
# withhold one constant label from the local store. It
# is redundant there — one VictoriaMetrics per swarm, so
# every series in it already belongs to this swarm — but
# a constant label multiplies no series, and it means
# what LEAVES and what STAYS have the same shape.
#
# Upstream is where it stops being redundant: that is the
# one hop where several swarms can land in one store, and
# samples that cannot name their swarm collide there
# exactly as hives collided here before per-hive
# receivers existed.
{
key = "swarm";
value = swarmDisplayName;
action = "upsert";
}
];
}
) hivePorts
# The swarm tier's own stamp: `swarm` and deliberately NO
# `hive`. A scraped swarm service belongs to the swarm, not to
# any one hive, so there is no honest value to put there — and
# an invented one (a sentinel, the local hive's name) would be
# queried as though it meant something.
// lib.optionalAttrs (cfg.scrapeTargets != { }) {
"resource/swarm".attributes = [
{
key = "swarm";
value = swarmDisplayName;
action = "upsert";
}
];
}
) hivePorts;
};
};
};