From 1af0138928592c5a0e39334c5b16e072a4ee0a22 Mon Sep 17 00:00:00 2001 From: atlas Date: Mon, 24 Aug 2026 16:59:01 +0200 Subject: [PATCH] fix(swarm-otel): tell the log store which field carries the message The journald receiver leaves the OTLP body empty and carries the entry as a map of journal fields, so VictoriaLogs had no message to index and wrote a placeholder into _msg on every record. Ingest returned 200, every field was present, and a plain search for a line sitting in the store found nothing. _msg_field names the field that holds the text. _stream_fields is the difference between one stream for the whole host and one per unit per machine; both are set by journald itself and both are low-cardinality. --- nix/host-modules/swarm-otel.nix | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/nix/host-modules/swarm-otel.nix b/nix/host-modules/swarm-otel.nix index fddaf4c7..51993a92 100644 --- a/nix/host-modules/swarm-otel.nix +++ b/nix/host-modules/swarm-otel.nix @@ -852,8 +852,26 @@ in # wrong path and a nonsense path all answer 400. Only its log # distinguishes them — the real route complains about the # encoding, everything else says "unsupported path requested". + # 🔑 THE TWO QUERY PARAMETERS ARE NOT TUNING — without the + # first one this pipeline fills the store with records that + # cannot be searched by message text, which is the only + # reason to collect logs at all. + # + # The journald receiver leaves the OTLP *body* empty and + # carries the entry as a map of journal fields, so the store + # has no message to index and writes the literal placeholder + # `missing _msg field` into `_msg` on EVERY record. Nothing + # errors: ingest returns 200, the data is all present, and a + # plain search for a line that is sitting right there returns + # nothing. `_msg_field` tells the store which field carries + # the message. Measured, both with and without. + # + # `_stream_fields` is the difference between one enormous + # stream for the whole host and one per unit per machine — + # both low-cardinality, and both fields journald sets itself. "otlphttp/victorialogs".logs_endpoint = - "http://127.0.0.1:${toString vlCfg.port}/insert/opentelemetry/v1/logs"; + "http://127.0.0.1:${toString vlCfg.port}/insert/opentelemetry/v1/logs" + + "?_msg_field=MESSAGE&_stream_fields=_HOSTNAME,_SYSTEMD_UNIT"; }; # Moves this collector's self-metrics off the built-in