vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-otel's own logs go only to the store, so an upstream export failure no longer feeds itself

Browse source 
The logs/self pipeline shipped the collector's own journal to the upstream
exporter too. When the upstream failed, the collector logged the failure,
journald/self read it, and logs/self sent it upstream again: ~8 lines/s.

logs/self now exports only to otlphttp/victorialogs, and a filter processor
drops the collector's failure lines that name that exporter, so the store
exporter cannot rebuild the loop one hop later. Other collector lines,
including upstream failures, still reach the store.

On deploy, swarm-otel's store rows drop from ~8/s of error spam to its real
output.
This commit is contained in:
atlas 2026-10-20 00:13:25 +02:00 • committed by mara
commit 8d70afd93e
3 changed files with 52 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

28
nix/module-eval/swarm-otel-core.nix
View file

@ -37,6 +37,9 @@ let
deploy.bao.otelOidcClientKeyFile = "/etc/pki/bao-otel-oidc-key.pem";
deploy.victoriametrics.enable = false;
deploy.victorialogs.enable = false;
# An upstream makes `logExporterNames` longer than the store alone, which
# is what the self-journal case below needs to tell the two apart.
otel.endpoint = "https://upstream.example/otel";
};
# The collector beside authelia, reading its own OIDC secret out of the
@ -285,6 +288,31 @@ let
&& lib.elem "resource/self-journal" (lib.head self).processors
&& builtins.elem "resource/self-journal" (lib.attrNames s.processors);
}
{
# 🩸 The collector logs its own export failures to the journal that
# `journald/self` reads. Shipping that to the upstream feeds each
# failure back into the failing exporter. Presence controls: the
# upstream exporter exists and the host-journal pipeline does send to
# it, so an exporter-name typo cannot make the negative pass.
name = "the collector's own journal never reaches the upstream, and drops the store exporter's own failures";
ok =
let
s = otelSettings otelNoStores;
pipelines = lib.attrValues s.service.pipelines;
self = lib.filter (p: builtins.elem "journald/self" p.receivers) pipelines;
host = lib.filter (p: builtins.elem "journald" p.receivers) pipelines;
f = s.processors."filter/drop-store-export-failures";
in
s.exporters ? otlphttp
&& lib.length host == 1
&& lib.all (p: builtins.elem "otlphttp" p.exporters) host
&& lib.length self == 1
&& !(lib.any (p: builtins.elem "otlphttp" p.exporters) self)
&& lib.elem "filter/drop-store-export-failures" (lib.head self).processors
&& lib.length f.logs.log_record == 1
&& lib.hasInfix "Exporting failed" (lib.head f.logs.log_record)
&& lib.hasInfix "otlphttp/victorialogs" (lib.head f.logs.log_record);
}
];
in
runGroup "swarm-otel-core" cases