grafana: show which sources are shipping, not just that the store is up

The log-store board reports rows ingested, disk size, free disk and errors —
every one of them a fact about VictoriaLogs itself. None of them can say
whether a particular unit or host is contributing, so aggregate ingestion
reads healthy on host-tier units while a whole tier ships nothing, and there
was no way to tell those apart from a dashboard.

Add four panels to that board rather than a second one. Every other board here
is per service — agents, authelia, openbao, forge, queue, the two stores — so
a second board about the same service would have made an operator guess which
of the two answers their question.

Nothing in the new panels names a unit: both breakdowns discover their rows
from the data, so a source that starts shipping appears without an edit, and
one that never existed is simply not there.

The ungrouped total is a control, not a summary. An empty breakdown renders
the same whether the query is malformed or the source genuinely never shipped;
with the total beside it, nonzero-and-empty is a broken query and
zero-and-empty is an empty store. Being on one board buys a second reading it
could not have alone: that total and `Log rows ingested` are the same quantity
measured by querying and from the store's own metrics, so the two disagreeing
means rows arrived that no query can reach — which is the failure this
pipeline actually had once.

The two breakdowns are bargauges rather than tables. A `stats` query returns
one frame per group, and a table panel renders one frame at a time behind a
series picker; bargauge consumes multi-series natively, for the same reason
the timeseries beside it always did. The alternative was a table plus
labelsToFields plus merge plus organize — three assumptions where this needs
none, in a spot I cannot render to check.

Series are named by their label rather than `rows{_SYSTEMD_UNIT="x.service"}`,
which also cleans up the timeseries legend. The datasource supports
legendFormat; its query editor's own placeholder is `{{label}}`.

These are the first panels to query the logs datasource at all; the other
eight boards are prometheus, including this one until now, which reads
VictoriaLogs' self-metrics out of VictoriaMetrics. So `renderDashboard` grows
a `@logsDatasourceUid@` substitution alongside the metrics one. The binding it
points at already existed, and its comment claiming a dashboard panel named it
only becomes true with this commit.

The uid is unchanged, so existing links and bookmarks still resolve; only the
title widens to match what the board now covers.

The query model was read out of the plugin in the store rather than guessed:
`queryType` is one of hits/instant/logs/stats/statsRange, and Stats/StatsRange
are the two that consume `expr`. The stream field names come from the module
that builds them — swarm-otel.nix's `_stream_fields=_HOSTNAME,_SYSTEMD_UNIT`.
The queries themselves are confirmed against the live store: mara ran the
by-unit one over seven days and it returned rows.

Refs #4084
This commit is contained in:
atlas 2026-09-07 23:15:09 +02:00 committed by mara
commit a204b5f457
2 changed files with 233 additions and 8 deletions

View file

@ -39,15 +39,16 @@ let
# free to drift into a panel that renders empty rather than erroring.
logsDatasourceUid = "swarm-victorialogs";
# The shipped dashboards carry `@datasourceUid@` where a real deployment needs
# the uid above. They are substituted here rather than committed with the
# literal so the single binding stays single.
# The shipped dashboards carry `@datasourceUid@` / `@logsDatasourceUid@` where
# a real deployment needs the uids above. They are substituted here rather
# than committed with the literals so the single binding stays single.
renderDashboard =
name:
pkgs.writeText name (
builtins.replaceStrings [ "@datasourceUid@" ] [ datasourceUid ] (
builtins.readFile (./swarm-grafana/dashboards + "/${name}")
)
builtins.replaceStrings
[ "@datasourceUid@" "@logsDatasourceUid@" ]
[ datasourceUid logsDatasourceUid ]
(builtins.readFile (./swarm-grafana/dashboards + "/${name}"))
);
# Grafana's file provider wants a DIRECTORY to scan, so the rendered files are