e92f118bda
swarm-metrics reads the swarm's VictoriaMetrics from inside an agent
container: query, query-range, series, labels and label-values, each one
GET against the store's Prometheus read API through the gateway, printing
the store's JSON. It authenticates the way swarm-logs does, with the
agent's own queue secret read from bao under the agent's leaf, sent as
a swarm-agent bearer.
On the metrics store's host the read API's auth map gains an agent arm
ahead of the Grafana one. It routes to swarm-metrics-agent-auth, a
swarm-logs-auth instance logged in under the existing swarm-metrics-auth
leaf, whose bao grant now includes swarm/agents/+/queue read. The check is
rate-limited per address like the log store's
(deploy.victoriametrics.agentAuth{Rate,Burst,VerdictTtl}).
nix/agent-modules/metrics.nix installs the wrapped binary when the hive
forwards HIVE_AGENT_METRICS_URL (from swarm.victoriametrics.domain) and a
store address, the same gate as swarm-logs. Adds the base:swarm-metrics
skill and the CLI reference with its freshness check.
Fixes #5395
2.3 KiB
| name | description |
|---|---|
| swarm-metrics | Query the swarm metrics store with `swarm-metrics` when you need to know whether a service's metrics reach VictoriaMetrics, which metric names or labels exist, or what a counter or gauge did over a window. Covers how to match dotted OTLP metric names, how to bound a query, and how to tell an empty answer from a refused request. Use this before running `swarm-metrics` for the first time in a session, and before you report a metric as missing. |
swarm-metrics
swarm-metrics reads the swarm's VictoriaMetrics through the gateway as you,
the agent. Every verb prints the store's JSON answer, so pipe it into jq.
Find out what exists first
Metric names from OTLP keep their dots (jobq.nodes,
hyperhive.vcs.commit.count), and so do OTLP attribute labels
(service.name). Match a dotted metric by __name__, and quote a dotted
label name:
$ swarm-metrics label-values __name__ --match '{__name__=~"jobq\\..*"}' | jq -r '.data[]'
$ swarm-metrics label-values service.name | jq -r '.data[]'
$ swarm-metrics series '{"service.name"="nats"}' --start -1h --limit 20
series, labels and label-values cover the last day from 00:00 UTC
unless you pass --start/--end. Use --limit on a broad selector.
Query
$ swarm-metrics query 'count by (__name__) ({__name__=~"hyperhive\\.vcs\\..*"})'
$ swarm-metrics query-range 'sum(rate({__name__="jobq.nodes"}[5m]))' --start -6h --step 5m
Times take RFC 3339, a unix timestamp or a relative duration (-1h).
query-range requires --start, so a range is always bounded. Inside the
expression, a regex needs its dots escaped twice (\\.), once for the
PromQL string and once for the regex.
Telling "no series" from "refused"
- No matching series: exit 0, and
.data.result(or.data) is[]. - Refused (auth, bad expression, rate limit): non-zero exit, with the HTTP status and the body on stderr. A 403 is the gateway's rate limit.
Before you report a metric missing, run a control
An empty answer fits a wrong name, a missing label value, or a window too
narrow just as well as a metric that never arrived. Query something that must
exist, such as label-values service.name, and widen --start. If the
control is also empty, the query is wrong, not the store.