otel: collectors drop their own export-failure lines instead of shipping them
This commit is contained in:
parent
f5e99fe248
commit
3fa926d7d6
1 changed files with 115 additions and 3 deletions
21
nix/collector-own-export-failures.nix
Normal file
21
nix/collector-own-export-failures.nix
Normal file
|
|
@ -0,0 +1,21 @@
|
|||
# The `filter` processor every collector's logs pipeline carries so that a
|
||||
# collector never ships its own export-failure lines.
|
||||
#
|
||||
# A collector that cannot reach its exporter logs one line per failed batch
|
||||
# (`Exporting failed. Will retry the request after interval.`, `Exporting
|
||||
# failed. Dropping data.`, and the 503 body `sending queue is full`) into the
|
||||
# journal its own journald receiver reads. Shipping those lines is more
|
||||
# export traffic during the outage that caused them.
|
||||
#
|
||||
# Only the collector's own unit, and only exporter failures: its startup,
|
||||
# config and receiver errors still ship. The failure stays visible as the
|
||||
# collector's `otelcol_exporter_send_failed_*` metrics.
|
||||
#
|
||||
# `body` is the journal entry as a map. A record whose body is not a map
|
||||
# raises an OTTL error, which `ignore` turns into "kept".
|
||||
{
|
||||
error_mode = "ignore";
|
||||
logs.log_record = [
|
||||
''body["_SYSTEMD_UNIT"] == "opentelemetry-collector.service" and IsMatch(body["MESSAGE"], "Exporting failed|sending queue is full")''
|
||||
];
|
||||
}
|
||||
Loading…
Reference in a new issue