diff --git a/nix/host-modules/swarm-otel.nix b/nix/host-modules/swarm-otel.nix index fddaf4c7..51993a92 100644 --- a/nix/host-modules/swarm-otel.nix +++ b/nix/host-modules/swarm-otel.nix @@ -852,8 +852,26 @@ in # wrong path and a nonsense path all answer 400. Only its log # distinguishes them — the real route complains about the # encoding, everything else says "unsupported path requested". + # 🔑 THE TWO QUERY PARAMETERS ARE NOT TUNING — without the + # first one this pipeline fills the store with records that + # cannot be searched by message text, which is the only + # reason to collect logs at all. + # + # The journald receiver leaves the OTLP *body* empty and + # carries the entry as a map of journal fields, so the store + # has no message to index and writes the literal placeholder + # `missing _msg field` into `_msg` on EVERY record. Nothing + # errors: ingest returns 200, the data is all present, and a + # plain search for a line that is sitting right there returns + # nothing. `_msg_field` tells the store which field carries + # the message. Measured, both with and without. + # + # `_stream_fields` is the difference between one enormous + # stream for the whole host and one per unit per machine — + # both low-cardinality, and both fields journald sets itself. "otlphttp/victorialogs".logs_endpoint = - "http://127.0.0.1:${toString vlCfg.port}/insert/opentelemetry/v1/logs"; + "http://127.0.0.1:${toString vlCfg.port}/insert/opentelemetry/v1/logs" + + "?_msg_field=MESSAGE&_stream_fields=_HOSTNAME,_SYSTEMD_UNIT"; }; # Moves this collector's self-metrics off the built-in