grafana: move logLevelRules implementation notes to docs

The 41-line comment on logLevelRules tripped the comment-block lint
(30-line max). Move the detailed walkthrough (why each rule shape is
what it is, the three silent-failure modes, the query a level button
emits) into docs/scheduler/observability.md's existing 'Log severity'
section, which already covered the mapping at a higher level. The nix
comment now carries the short why/contract and points at the doc for
the full detail — no information dropped, just relocated.
This commit is contained in:
atlas 2026-09-20 16:37:30 +02:00 committed by mara
commit da1f80c2ef
2 changed files with 49 additions and 46 deletions

View file

@ -260,19 +260,49 @@ while lines that never had one signal nothing of the sort.
The stored field is `severity_text`, and Grafana's log-level buttons filter on
a field called `level` — a name no row here carries. That is not something the
pipeline can fix: VictoriaLogs' OTLP ingester chooses the stored name itself,
and its ingest parameters have no option to rename it.
pipeline can fix: VictoriaLogs' OTLP ingester names the stored field
`severity_text` unconditionally (v1.52.0, `app/vlinsert/opentelemetry/pb.go`),
and its ingest parameters have no `_level_field` sibling to rename it with.
So the datasource is told instead. `nix/host-modules/swarm-grafana.nix`
provisions the VictoriaLogs datasource with a `logLevelRules` entry per
severity in the table above, each matching `severity_text` against one of
those values — which is the plugin's way of saying "the level lives in this
field". A level button then filters on `level` **or** the matching
`severity_text`, and the store answers the second half.
field". `logLevelRules` is the datasource plugin's own jsonData key and the
only level-related one it has — there is no single "the level lives in field
X" string to set, so mapping every level takes one rule each. A level button
then filters on `level` **or** the matching `severity_text`, and the store
answers the second half. Each enabled rule appends an `OR
<field>:<op>"<value>"` term to the query a level button emits, beside the
`level:…` term that matches nothing here. Clicking "info" goes from
`Unspecified` is deliberately left out of those rules, so rows with no
severity stay unfiltered by every level button and keep showing up in the
panel that counts them.
```
level:contains_common_case("info","information","informational","notice")
```
to that OR `severity_text:="INFO"`, which our rows do match.
`Unspecified` is deliberately left out of those rules. It is VictoriaLogs' own
rendering of an absent severity, and it is what the **Log rows with no
severity** panel counts — giving it a level would dress the missing data up
as a colour and retire the instrument that measures it.
#### Three ways to get a `logLevelRules` entry wrong, all of them silent
Grafana accepts any jsonData it does not recognise, so a bad rule provisions
cleanly and the affected level button goes on returning zero rows:
- `enabled` must be literally `true`, not merely not-false. The query
builder keeps rules on `rule.enabled` being truthy while the row colouring
path keeps them on `!== false`, so an omitted flag colours rows correctly
and leaves the buttons broken — working in the half nobody is looking at.
- `level` must be a canonical Grafana `LogLevel` value. The builder groups
rules by it and only ever looks up `critical`, `error`, `warning`, `info`,
`debug`, `trace`; `warn` and `fatal` are enum aliases that resolve to
other spellings and match no group.
- `value` is compared with `===`, so it must be the severity text exactly as
stored: the uppercase OpenTelemetry short names `overwrite_text` writes in
`nix/journald-severity.nix`, not that table's lowercase keys.
## Host-emitted container-resource metrics (hive-c0re)