Two findings on one panel, both from mara.
Form: 24 label series drawn as overlapping timeseries lines is
unreadable by construction. The panel is a current-count ranking, not a
time evolution, so it becomes a horizontal bar gauge on an instant
sort_desc query -- identity on the axis, magnitude in the bar, and a
fixed colour because colour was carrying no information that the axis
was not already carrying.
Title: the metric counts open AND closed issues. Measured, not assumed
-- the largest single label reads 1619 while gitea_issues_open reads
117, and no label can outnumber the set it is a subset of. The panel had
been asserting the opposite since it was written.
mara on the PR: "many hint texts are incorrect, most hives dont work on
hyperhive itself". Four panel descriptions asserted local practice as if
it were general:
Releases "expected to sit at zero on a hive that ships from
main" -- a claim about how THIS project releases
Automation surface "every hive registering the same webhook has bitten
this forge before" -- a war story from this
deployment, meaningless on any other
Organizations "near-constant by design" -- assumes our org layout
Issues open/closed "usually a bulk triage, not a productive afternoon"
-- a claim about our working rhythm
Each now describes what the metric IS and what a CHANGE means, and says
outright where the panel cannot know: whether zero releases is normal
depends on how a hive's projects ship, and a vertical step in closed is
some batch operation that this panel cannot distinguish from a burst of
real work.
The board title is NOT an instance of this and is unchanged -- all seven
dashboards use the 'hyperhive - <thing>' product namespace, checked
rather than assumed.
The forge restarted (new forgejo PID at 12:36), so it re-read [metrics]
and gitea_issues_by_label now exists -- 24 series carrying real label
names, read back out of the store rather than inferred from the metric
name appearing in a label-values list.
The panel shipped deliberately empty with EMPTY UNTIL THE FORGE RESTARTS
in its title and a matching noValue, because an empty graph and an
undeployed metric look identical. That warning is now false, and a stale
warning is worse than none: it tells a reader the panel cannot have data
at the moment it does.
Keeps the operational half rather than deleting it -- the restart
requirement is a fact about the NEXT metric someone enables, so it moves
into the description as a note and into noValue, where it fires only when
the series really is missing.
Adds the fourth per-service board. Unlike its three siblings it answers
"what does the forge CONTAIN", not "is the forge healthy" -- forgejo's
exporter publishes a census of stored objects and no request, error or
latency series at all.
That distinction is the board's own description rather than something a
reader has to infer, because a fourth per-service board that silently
answers a different question than the other three is worse than no board:
someone checks it during an incident and reads a flat line as calm.
Panels: repositories, users, orgs, issues open/closed, releases, the
open-vs-closed pair over time, repos-and-users together (they move in
lockstep when an agent is created, so divergence is the signal),
discussion volume, and the webhook/hooktask pair.
One panel ships deliberately EMPTY: issues-by-label. The metric is
enabled in the forge's config but forgejo reads its [metrics] section at
startup, so the series does not exist until that process restarts -- and
a config merge alone does not restart it. The panel title says so, the
description says how to tell "not deployed" from "no labelled issues",
and `noValue` says it again in the graph itself. An empty panel with no
explanation is the failure this board is trying not to be.
Gated: every panel's metric confirmed to exist in the store, with a
bogus name checked absent in the same run so the check can fail, plus a
known-positive control against the label-values query itself -- a
download that silently returned nothing would otherwise report every
metric missing. Gate kept as verify-3494-forge-board.sh.
Two more per-service boards, cut from the withdrawn combined services
board: the log store (4 panels) and the metrics store (5).
The combined board was one page carrying panels for four different
services; the seams were already there, so this is a re-cut rather than
new authoring. Each board's subject is one service, and the services
page that remains will carry only panels whose subject is the *set*.
Both were imported by the operator before being committed here, so the
queries are known to draw against the live stores rather than only
parsing.
The pending-bytes panel described itself as the leading indicator for
the slow-consumer stat 'beside it', and then sat alone in its own row
with eighteen grid units of empty space. A description that tells the
reader to look beside it, where nothing is, is worse than no
description.
Slow consumers and pending bytes are now adjacent in the top row, and
the two JetStream panels share the second. No gaps.
Split out of the combined swarm-services board, which mara declined as
multiple dashboards in one trench coat. Its 17 panels partition exactly
by subject -- 3 services / 5 metrics store / 5 queue / 4 log store, with
no panel belonging to two -- so this is a re-cut of verified panels, not
new authoring.
Seven panels: connections, subscriptions, slow consumers and JetStream
API errors as stats, messages over the range, plus pending bytes (the
leading indicator for a slow consumer) and JetStream store size against
its configured limit (the only capacity question the queue has).
Connections is red at zero, unlike every other count here: three things
hold a connection at all times, so an empty count means the queue is
refusing everyone -- and a callout denial reaches a client as a timeout
rather than an error, which makes this often the first visible sign.
The swarm-services board was mostly authelia, so it becomes its own page
and is trimmed out of that one (mara, #3494). Eleven panels: uptime,
authentications and failures, authorization decisions, requests and
verdicts by status code, request and OIDC latency, and the three Go
process signals.
OIDC latency gets its own panel rather than being folded into the
general one because authelia keeps a separate duration family for it,
and every machine-to-machine credential in this swarm is minted through
those endpoints -- averaging the two produces a number describing
neither.
Counters are counted over the dashboard range rather than rated. At this
volume a rate window contains no requests, so rate() returns zero and
draws a flat line, which is indistinguishable from a broken query; the
quantile panels are worse, because a quantile over all-zero buckets is
NaN and renders empty rather than zero.
Dashboard 2 of the set the operator asked for: how Claude is being used
rather than which agent is using it, so the axes are model, effort, token
type and query source. The only deliberate overlap with the agents page
is the cost/token headline.
Every panel was run against the live store before this landed. Three
panels were dropped rather than shipped, because their label has exactly
one live value today and a page of single-bar charts is the same silent
failure as an empty one.
Turn count and turn length are PROXIES and say so in their descriptions:
nothing exports turn stats, so a session record stands in for a turn,
which holds because each turn runs a new claude process.
The operator asked for this when the dashboard was first reviewed and it was
deferred, not declined: the container metric family carried no hive label, so
selecting a hive emptied every container panel while the "All" default hid the
problem completely -- a `.*` matcher matches series where the label is absent.
That family now carries hive and swarm, verified against the live store rather
than inferred from the fix having merged: a hive selection returns the same 7
agents as the All default, at every window out to 168h, with a nonexistent
hive returning zero.
The agent list chains off the hive selection, so picking a hive narrows the
agent dropdown rather than leaving entries in it that resolve to nothing.
Grafana served no dashboards: only datasources were provisioned, while
the module header already claimed dashboards were. This ships the agents
dashboard as a file provider and makes that sentence true.
The datasource uid is bound once and substituted into the dashboard at
build time. Committing the literal would make the dashboard a second
speller of a name the datasource already owns, and the drift failure is
silent -- panels render empty rather than erroring.
The shipped copy drops the `DS` datasource variable: it exists so an
operator can pick a store on manual import, and a provisioned dashboard
must not ask.
