add swarm-metrics, an agent CLI for the metrics store
swarm-metrics reads the swarm's VictoriaMetrics from inside an agent
container: query, query-range, series, labels and label-values, each one
GET against the store's Prometheus read API through the gateway, printing
the store's JSON. It authenticates the way swarm-logs does, with the
agent's own queue secret read from bao under the agent's leaf, sent as
a swarm-agent bearer.
On the metrics store's host the read API's auth map gains an agent arm
ahead of the Grafana one. It routes to swarm-metrics-agent-auth, a
swarm-logs-auth instance logged in under the existing swarm-metrics-auth
leaf, whose bao grant now includes swarm/agents/+/queue read. The check is
rate-limited per address like the log store's
(deploy.victoriametrics.agentAuth{Rate,Burst,VerdictTtl}).
nix/agent-modules/metrics.nix installs the wrapped binary when the hive
forwards HIVE_AGENT_METRICS_URL (from swarm.victoriametrics.domain) and a
store address, the same gate as swarm-logs. Adds the base:swarm-metrics
skill and the CLI reference with its freshness check.
Fixes #5395
This commit is contained in:
parent
1e8c398132
commit
e92f118bda
30 changed files with 1417 additions and 88 deletions
50
claude-plugins/plugins/base/skills/swarm-metrics/SKILL.md
Normal file
50
claude-plugins/plugins/base/skills/swarm-metrics/SKILL.md
Normal file
|
|
@ -0,0 +1,50 @@
|
|||
---
|
||||
name: swarm-metrics
|
||||
description: Query the swarm metrics store with `swarm-metrics` when you need to know whether a service's metrics reach VictoriaMetrics, which metric names or labels exist, or what a counter or gauge did over a window. Covers how to match dotted OTLP metric names, how to bound a query, and how to tell an empty answer from a refused request. Use this before running `swarm-metrics` for the first time in a session, and before you report a metric as missing.
|
||||
---
|
||||
|
||||
# swarm-metrics
|
||||
|
||||
`swarm-metrics` reads the swarm's VictoriaMetrics through the gateway as you,
|
||||
the agent. Every verb prints the store's JSON answer, so pipe it into `jq`.
|
||||
|
||||
## Find out what exists first
|
||||
|
||||
Metric names from OTLP keep their dots (`jobq.nodes`,
|
||||
`hyperhive.vcs.commit.count`), and so do OTLP attribute labels
|
||||
(`service.name`). Match a dotted metric by `__name__`, and quote a dotted
|
||||
label name:
|
||||
|
||||
```console
|
||||
$ swarm-metrics label-values __name__ --match '{__name__=~"jobq\\..*"}' | jq -r '.data[]'
|
||||
$ swarm-metrics label-values service.name | jq -r '.data[]'
|
||||
$ swarm-metrics series '{"service.name"="nats"}' --start -1h --limit 20
|
||||
```
|
||||
|
||||
`series`, `labels` and `label-values` cover the last day from 00:00 UTC
|
||||
unless you pass `--start`/`--end`. Use `--limit` on a broad selector.
|
||||
|
||||
## Query
|
||||
|
||||
```console
|
||||
$ swarm-metrics query 'count by (__name__) ({__name__=~"hyperhive\\.vcs\\..*"})'
|
||||
$ swarm-metrics query-range 'sum(rate({__name__="jobq.nodes"}[5m]))' --start -6h --step 5m
|
||||
```
|
||||
|
||||
Times take RFC 3339, a unix timestamp or a relative duration (`-1h`).
|
||||
`query-range` requires `--start`, so a range is always bounded. Inside the
|
||||
expression, a regex needs its dots escaped twice (`\\.`), once for the
|
||||
PromQL string and once for the regex.
|
||||
|
||||
## Telling "no series" from "refused"
|
||||
|
||||
- **No matching series**: exit 0, and `.data.result` (or `.data`) is `[]`.
|
||||
- **Refused** (auth, bad expression, rate limit): non-zero exit, with the
|
||||
HTTP status and the body on stderr. A 403 is the gateway's rate limit.
|
||||
|
||||
## Before you report a metric missing, run a control
|
||||
|
||||
An empty answer fits a wrong name, a missing label value, or a window too
|
||||
narrow just as well as a metric that never arrived. Query something that must
|
||||
exist, such as `label-values service.name`, and widen `--start`. If the
|
||||
control is also empty, the query is wrong, not the store.
|
||||
Loading…
Reference in a new issue