Watch
0
0
Fork
You've already forked hyperhive
0

add swarm-metrics, an agent CLI for the metrics store

swarm-metrics reads the swarm's VictoriaMetrics from inside an agent
container: query, query-range, series, labels and label-values, each one
GET against the store's Prometheus read API through the gateway, printing
the store's JSON. It authenticates the way swarm-logs does, with the
agent's own queue secret read from bao under the agent's leaf, sent as
a swarm-agent bearer.

On the metrics store's host the read API's auth map gains an agent arm
ahead of the Grafana one. It routes to swarm-metrics-agent-auth, a
swarm-logs-auth instance logged in under the existing swarm-metrics-auth
leaf, whose bao grant now includes swarm/agents/+/queue read. The check is
rate-limited per address like the log store's
(deploy.victoriametrics.agentAuth{Rate,Burst,VerdictTtl}).

nix/agent-modules/metrics.nix installs the wrapped binary when the hive
forwards HIVE_AGENT_METRICS_URL (from swarm.victoriametrics.domain) and a
store address, the same gate as swarm-logs. Adds the base:swarm-metrics
skill and the CLI reference with its freshness check.

Fixes #5395
This commit is contained in:
atlas 2026-10-17 00:55:46 +02:00 • committed by mara
commit e92f118bda
30 changed files with 1417 additions and 88 deletions

View file

@ -0,0 +1,50 @@
---
name: swarm-metrics
description: Query the swarm metrics store with `swarm-metrics` when you need to know whether a service's metrics reach VictoriaMetrics, which metric names or labels exist, or what a counter or gauge did over a window. Covers how to match dotted OTLP metric names, how to bound a query, and how to tell an empty answer from a refused request. Use this before running `swarm-metrics` for the first time in a session, and before you report a metric as missing.
---
# swarm-metrics
`swarm-metrics` reads the swarm's VictoriaMetrics through the gateway as you,
the agent. Every verb prints the store's JSON answer, so pipe it into `jq`.
## Find out what exists first
Metric names from OTLP keep their dots (`jobq.nodes`,
`hyperhive.vcs.commit.count`), and so do OTLP attribute labels
(`service.name`). Match a dotted metric by `__name__`, and quote a dotted
label name:
```console
$ swarm-metrics label-values __name__ --match '{__name__=~"jobq\\..*"}' | jq -r '.data[]'
$ swarm-metrics label-values service.name | jq -r '.data[]'
$ swarm-metrics series '{"service.name"="nats"}' --start -1h --limit 20
```
`series`, `labels` and `label-values` cover the last day from 00:00 UTC
unless you pass `--start`/`--end`. Use `--limit` on a broad selector.
## Query
```console
$ swarm-metrics query 'count by (__name__) ({__name__=~"hyperhive\\.vcs\\..*"})'
$ swarm-metrics query-range 'sum(rate({__name__="jobq.nodes"}[5m]))' --start -6h --step 5m
```
Times take RFC 3339, a unix timestamp or a relative duration (`-1h`).
`query-range` requires `--start`, so a range is always bounded. Inside the
expression, a regex needs its dots escaped twice (`\\.`), once for the
PromQL string and once for the regex.
## Telling "no series" from "refused"
- **No matching series**: exit 0, and `.data.result` (or `.data`) is `[]`.
- **Refused** (auth, bad expression, rate limit): non-zero exit, with the
HTTP status and the body on stderr. A 403 is the gateway's rate limit.
## Before you report a metric missing, run a control
An empty answer fits a wrong name, a missing label value, or a window too
narrow just as well as a metric that never arrived. Query something that must
exist, such as `label-values service.name`, and widen `--start`. If the
control is also empty, the query is wrong, not the store.