diff --git a/docs/scheduler/observability.md b/docs/scheduler/observability.md index d56a5e44..5b8e7cf4 100644 --- a/docs/scheduler/observability.md +++ b/docs/scheduler/observability.md @@ -260,19 +260,49 @@ while lines that never had one signal nothing of the sort. The stored field is `severity_text`, and Grafana's log-level buttons filter on a field called `level` — a name no row here carries. That is not something the -pipeline can fix: VictoriaLogs' OTLP ingester chooses the stored name itself, -and its ingest parameters have no option to rename it. +pipeline can fix: VictoriaLogs' OTLP ingester names the stored field +`severity_text` unconditionally (v1.52.0, `app/vlinsert/opentelemetry/pb.go`), +and its ingest parameters have no `_level_field` sibling to rename it with. So the datasource is told instead. `nix/host-modules/swarm-grafana.nix` provisions the VictoriaLogs datasource with a `logLevelRules` entry per severity in the table above, each matching `severity_text` against one of those values — which is the plugin's way of saying "the level lives in this -field". A level button then filters on `level` **or** the matching -`severity_text`, and the store answers the second half. +field". `logLevelRules` is the datasource plugin's own jsonData key and the +only level-related one it has — there is no single "the level lives in field +X" string to set, so mapping every level takes one rule each. A level button +then filters on `level` **or** the matching `severity_text`, and the store +answers the second half. Each enabled rule appends an `OR +:""` term to the query a level button emits, beside the +`level:…` term that matches nothing here. Clicking "info" goes from -`Unspecified` is deliberately left out of those rules, so rows with no -severity stay unfiltered by every level button and keep showing up in the -panel that counts them. +``` +level:contains_common_case("info","information","informational","notice") +``` + +to that OR `severity_text:="INFO"`, which our rows do match. + +`Unspecified` is deliberately left out of those rules. It is VictoriaLogs' own +rendering of an absent severity, and it is what the **Log rows with no +severity** panel counts — giving it a level would dress the missing data up +as a colour and retire the instrument that measures it. + +#### Three ways to get a `logLevelRules` entry wrong, all of them silent + +Grafana accepts any jsonData it does not recognise, so a bad rule provisions +cleanly and the affected level button goes on returning zero rows: + +- `enabled` must be literally `true`, not merely not-false. The query + builder keeps rules on `rule.enabled` being truthy while the row colouring + path keeps them on `!== false`, so an omitted flag colours rows correctly + and leaves the buttons broken — working in the half nobody is looking at. +- `level` must be a canonical Grafana `LogLevel` value. The builder groups + rules by it and only ever looks up `critical`, `error`, `warning`, `info`, + `debug`, `trace`; `warn` and `fatal` are enum aliases that resolve to + other spellings and match no group. +- `value` is compared with `===`, so it must be the severity text exactly as + stored: the uppercase OpenTelemetry short names `overwrite_text` writes in + `nix/journald-severity.nix`, not that table's lowercase keys. ## Host-emitted container-resource metrics (hive-c0re) diff --git a/nix/host-modules/swarm-grafana.nix b/nix/host-modules/swarm-grafana.nix index aaa301bd..5e6ac674 100644 --- a/nix/host-modules/swarm-grafana.nix +++ b/nix/host-modules/swarm-grafana.nix @@ -41,46 +41,19 @@ let logsDatasourceUid = "swarm-victorialogs"; # Which field carries the log level, told to the READER because the writer - # cannot be told. Grafana's log-level buttons filter on a field called - # `level`, and no row in this store has one: VictoriaLogs' OTLP ingester - # names the stored field `severity_text` unconditionally (v1.52.0, - # `app/vlinsert/opentelemetry/pb.go`) and its ingest parameters have no - # `_level_field` sibling to rename it with. So the mapping is made here. + # cannot be told: VictoriaLogs' OTLP ingester names the stored field + # `severity_text` unconditionally, with no option to rename it, but + # Grafana's log-level buttons filter on a field called `level`. So the + # mapping is made here, one `logLevelRules` entry per level below. # - # `logLevelRules` is the datasource plugin's own jsonData key and the only - # level-related one it has — there is no "the level lives in field X" - # string to set. A rule names its own field, so saying it takes one rule - # per level. - # - # 🔑 Each enabled rule appends an `OR :""` term to the - # query a level button emits, beside the `level:…` term that matches - # nothing here. Clicking "info" goes from - # - # level:contains_common_case("info","information","informational","notice") - # - # to that OR `severity_text:="INFO"`, which our rows do match. - # - # ⚠️ Three ways to get a rule wrong, all of them SILENT — Grafana accepts - # any jsonData it does not recognise, so a bad rule provisions cleanly and - # the buttons go on returning zero: - # - # - `enabled` must be literally `true`, not merely not-false. The query - # builder keeps rules on `rule.enabled` being truthy while the row - # colouring path keeps them on `!== false`, so an omitted flag colours - # rows correctly and leaves the buttons broken — working in the half - # nobody is looking at. - # - `level` must be a CANONICAL Grafana `LogLevel` value. The builder - # groups rules by it and only ever looks up `critical`, `error`, - # `warning`, `info`, `debug`, `trace`; `warn` and `fatal` are enum - # aliases that resolve to other spellings and match no group. - # - `value` is compared with `===`, so it is the severity text exactly as - # STORED: the uppercase OpenTelemetry short names `overwrite_text` - # writes in ../journald-severity.nix, not that table's lowercase keys. - # - # `Unspecified` is deliberately unmapped. It is VictoriaLogs' own rendering - # of an absent severity, and it is what the logs dashboard's "no severity" - # panel counts — giving it a level would dress the missing data up as a - # colour and retire the instrument that measures it. + # 🔑 A rule wrong in any of three ways provisions cleanly and its level + # button silently returns zero rows — `enabled` must be literally `true`, + # `level` must be a canonical Grafana `LogLevel` (not an alias like `warn` + # or `fatal`), and `value` must match ../journald-severity.nix's uppercase + # STORED text, not its lowercase keys. `Unspecified` is deliberately left + # unmapped so the "no severity" panel keeps counting it. Full walkthrough + # (why each rule shape is what it is, the query a level button actually + # emits): docs/scheduler/observability.md#why-explores-level-buttons-need-the-datasource-told logLevelRules = let fromSeverityText = value: level: {