vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

add swarm-metrics, an agent CLI for the metrics store

Browse source 
swarm-metrics reads the swarm's VictoriaMetrics from inside an agent
container: query, query-range, series, labels and label-values, each one
GET against the store's Prometheus read API through the gateway, printing
the store's JSON. It authenticates the way swarm-logs does, with the
agent's own queue secret read from bao under the agent's leaf, sent as
a swarm-agent bearer.

On the metrics store's host the read API's auth map gains an agent arm
ahead of the Grafana one. It routes to swarm-metrics-agent-auth, a
swarm-logs-auth instance logged in under the existing swarm-metrics-auth
leaf, whose bao grant now includes swarm/agents/+/queue read. The check is
rate-limited per address like the log store's
(deploy.victoriametrics.agentAuth{Rate,Burst,VerdictTtl}).

nix/agent-modules/metrics.nix installs the wrapped binary when the hive
forwards HIVE_AGENT_METRICS_URL (from swarm.victoriametrics.domain) and a
store address, the same gate as swarm-logs. Adds the base:swarm-metrics
skill and the CLI reference with its freshness check.

Fixes #5395
This commit is contained in:
atlas 2026-10-17 00:55:46 +02:00 • committed by mara
commit e92f118bda
30 changed files with 1417 additions and 88 deletions
Download patch file Download diff file Expand all files Collapse all files

1
.prettierignore
View file

@ -8,6 +8,7 @@ docs/tools/hivectl-cli.md
docs/tools/swarmctl-cli.md docs/tools/swarmctl-cli.md
docs/tools/forge-cli.md docs/tools/forge-cli.md
docs/tools/swarm-logs-cli.md docs/tools/swarm-logs-cli.md
docs/tools/swarm-metrics-cli.md
# Files with multi-line list-item continuations that prettier strips to col 0. # Files with multi-line list-item continuations that prettier strips to col 0.
# prettier's `proseWrap: "preserve"` prevents prose reflow but not list-item # prettier's `proseWrap: "preserve"` prevents prose reflow but not list-item

3
CLAUDE.md
View file

@ -138,10 +138,11 @@ Swarm-level:
token against the store; shared by `swarm-nats-auth` and token against the store; shared by `swarm-nats-auth` and
`swarm-logs-auth`. `swarm-logs-auth`.
- **`swarm-logs-auth/`** — nginx `auth_request` responder for agent access - **`swarm-logs-auth/`** — nginx `auth_request` responder for agent access
to the log store. to the log store, and to the metrics store as `swarm-metrics-agent-auth`.
- **`swarm-metrics-auth/`** — nginx `auth_request` responder for Grafana's - **`swarm-metrics-auth/`** — nginx `auth_request` responder for Grafana's
bearer on the metrics store's query API. bearer on the metrics store's query API.
- **`swarm-logs/`** — agent CLI: a LogsQL query in, log lines out. - **`swarm-logs/`** — agent CLI: a LogsQL query in, log lines out.
- **`swarm-metrics/`** — agent CLI: a MetricsQL query in, the store's JSON out.
- **`swarm-matrix-client/`** — appservice calls against the swarm - **`swarm-matrix-client/`** — appservice calls against the swarm
homeserver, shared by `swarm-controller` and `swarm-matrix-ctl`. homeserver, shared by `swarm-controller` and `swarm-matrix-ctl`.
- **`swarm-matrix-ctl/`** — the one binary run inside - **`swarm-matrix-ctl/`** — the one binary run inside

13
Cargo.lock generated
View file

@ -4944,6 +4944,19 @@ dependencies = [
"tracing-subscriber", "tracing-subscriber",
] ]
[[package]]
name = "swarm-metrics"
version = "0.1.0"
dependencies = [
"anyhow",
"clap",
"clap-markdown",
"reqwest",
"swarm-queue-client",
"swarm-secret-client",
"tokio",
]
[[package]] [[package]]
name = "swarm-metrics-auth" name = "swarm-metrics-auth"
version = "0.1.0" version = "0.1.0"

1
Cargo.toml
View file

@ -38,6 +38,7 @@ members = [
"swarm-queue-client", "swarm-queue-client",
"swarm-review-classifier", "swarm-review-classifier",
"swarm-logs", "swarm-logs",
"swarm-metrics",
"swarm-logs-auth", "swarm-logs-auth",
"swarm-metrics-auth", "swarm-metrics-auth",
"swarm-secret-client", "swarm-secret-client",

50
claude-plugins/plugins/base/skills/swarm-metrics/SKILL.md Normal file
View file

@ -0,0 +1,50 @@
---
name: swarm-metrics
description: Query the swarm metrics store with `swarm-metrics` when you need to know whether a service's metrics reach VictoriaMetrics, which metric names or labels exist, or what a counter or gauge did over a window. Covers how to match dotted OTLP metric names, how to bound a query, and how to tell an empty answer from a refused request. Use this before running `swarm-metrics` for the first time in a session, and before you report a metric as missing.
---
# swarm-metrics
`swarm-metrics` reads the swarm's VictoriaMetrics through the gateway as you,
the agent. Every verb prints the store's JSON answer, so pipe it into `jq`.
## Find out what exists first
Metric names from OTLP keep their dots (`jobq.nodes`,
`hyperhive.vcs.commit.count`), and so do OTLP attribute labels
(`service.name`). Match a dotted metric by `__name__`, and quote a dotted
label name:
```console
$ swarm-metrics label-values __name__ --match '{__name__=~"jobq\\..*"}' | jq -r '.data[]'
$ swarm-metrics label-values service.name | jq -r '.data[]'
$ swarm-metrics series '{"service.name"="nats"}' --start -1h --limit 20
```
`series`, `labels` and `label-values` cover the last day from 00:00 UTC
unless you pass `--start`/`--end`. Use `--limit` on a broad selector.
## Query
```console
$ swarm-metrics query 'count by (__name__) ({__name__=~"hyperhive\\.vcs\\..*"})'
$ swarm-metrics query-range 'sum(rate({__name__="jobq.nodes"}[5m]))' --start -6h --step 5m
```
Times take RFC 3339, a unix timestamp or a relative duration (`-1h`).
`query-range` requires `--start`, so a range is always bounded. Inside the
expression, a regex needs its dots escaped twice (`\\.`), once for the
PromQL string and once for the regex.
## Telling "no series" from "refused"
- **No matching series**: exit 0, and `.data.result` (or `.data`) is `[]`.
- **Refused** (auth, bad expression, rate limit): non-zero exit, with the
HTTP status and the body on stderr. A 403 is the gateway's rate limit.
## Before you report a metric missing, run a control
An empty answer fits a wrong name, a missing label value, or a window too
narrow just as well as a metric that never arrived. Query something that must
exist, such as `label-values service.name`, and widen `--start`. If the
control is also empty, the query is wrong, not the store.

2
docs/process/gotchas.md
View file

@ -83,7 +83,7 @@ With both set, systemd creates the runtime directory under `/run/private/`
nginx's `connect()` to a socket in it fails with `EACCES` whatever the nginx's `connect()` to a socket in it fails with `EACCES` whatever the
directory and socket modes are. A unit serving a socket to another user directory and socket modes are. A unit serving a socket to another user
runs as a static user instead (`swarm-controller`, `swarm-logs-auth`, runs as a static user instead (`swarm-controller`, `swarm-logs-auth`,
`swarm-metrics-auth`). `swarm-metrics-auth`, `swarm-metrics-agent-auth`).
`module-eval-core-toggle` fails on any `unix:` upstream of nginx owned by `module-eval-core-toggle` fails on any `unix:` upstream of nginx owned by
such a unit. such a unit.

4
docs/swarm/credentials.md
View file

@ -56,7 +56,7 @@ of the cell says how.
<!-- vale write-good.Passive = NO --> <!-- vale write-good.Passive = NO -->
| store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull | | store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull |
| ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `swarm/agents/<agent>/matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again | | `swarm/agents/<agent>/matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again |
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | ✅ a secondary that didn't come up at start, or whose token the homeserver rejected, is read from the store again two minutes later, then at doubling waits up to sixteen minutes, without restarting the daemon or its other accounts; one that's up and then rejected ends its sync loop, the daemon exits and its timer restarts it | | `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | ✅ a secondary that didn't come up at start, or whose token the homeserver rejected, is read from the store again two minutes later, then at doubling waits up to sixteen minutes, without restarting the daemon or its other accounts; one that's up and then rejected ends its sync loop, the daemon exits and its timer restarts it |
| `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase for every account listed under the agent's `matrix/`, in a five-minute pass | the agent container itself, under the certificate its hive passed in, when `hive-matrix-daemon` brings the account up | ❌ never replaced: the account's server-side key backup opens only with the passphrase it was enabled with, so the pass writes only to an empty path | ❌ read when the account comes up; an account whose passphrase lands later turns backup on at the next daemon start | | `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase for every account listed under the agent's `matrix/`, in a five-minute pass | the agent container itself, under the certificate its hive passed in, when `hive-matrix-daemon` brings the account up | ❌ never replaced: the account's server-side key backup opens only with the passphrase it was enabled with, so the pass writes only to an empty path | ❌ read when the account comes up; an account whose passphrase lands later turns backup on at the next daemon start |
@ -65,7 +65,7 @@ of the cell says how.
| `swarm/controller/swarm-controller/forge/token` | the forge's `forgejo-swarm-controller-account`, in the `hive-forge` container, once, under the forge publisher's certificate with a write-only grant; it then revokes its earlier tokens | `swarm-controller`, under its own certificate, at start and on refusal | ❌ the forge mints it once and keeps only the token's name in the container, at `/var/lib/forgejo/swarm-controller-token-name`; deleting that file mints a new token on the unit's next start | ✅ the controller reads it at start, and in the background with backoff until it first finds it. When the forge answers 401 it reads it again and sends the request once more with the new token, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value | | `swarm/controller/swarm-controller/forge/token` | the forge's `forgejo-swarm-controller-account`, in the `hive-forge` container, once, under the forge publisher's certificate with a write-only grant; it then revokes its earlier tokens | `swarm-controller`, under its own certificate, at start and on refusal | ❌ the forge mints it once and keeps only the token's name in the container, at `/var/lib/forgejo/swarm-controller-token-name`; deleting that file mints a new token on the unit's next start | ✅ the controller reads it at start, and in the background with backoff until it first finds it. When the forge answers 401 it reads it again and sends the request once more with the new token, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value |
| `swarm/controller/swarm-controller/grafana/token` | `swarm-bao-grafana-controller-token` on Grafana's host, under Grafana's certificate: a token for the `swarm-controller` Viewer service account, minted with Grafana's admin password | `swarm-controller`, under its own certificate, on the first `/api/alerts` request and again when Grafana refuses it | ✅ a timer starts the unit again 10 minutes after each run; it keeps a stored token Grafana accepts, and mints a new one when Grafana answers 401 or the store holds none | ✅ the controller holds the token in memory; when Grafana answers 401 it reads the store once more and retries the request | | `swarm/controller/swarm-controller/grafana/token` | `swarm-bao-grafana-controller-token` on Grafana's host, under Grafana's certificate: a token for the `swarm-controller` Viewer service account, minted with Grafana's admin password | `swarm-controller`, under its own certificate, on the first `/api/alerts` request and again when Grafana refuses it | ✅ a timer starts the unit again 10 minutes after each run; it keeps a stored token Grafana accepts, and mints a new one when Grafana answers 401 or the store holds none | ✅ the controller holds the token in memory; when Grafana answers 401 it reads the store once more and retries the request |
| `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config; then `hive-agent-bao-identity` in the agent container, under the leaf it holds | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ✅ `hive-agent-bao-identity` re-reads it every 10 minutes and swaps a new leaf into `/run/hive-agent-bao-identity/`, which every store login in the container presents. It reads under the leaf it holds, so it renews only while that leaf is valid | | `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config; then `hive-agent-bao-identity` in the agent container, under the leaf it holds | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ✅ `hive-agent-bao-identity` re-reads it every 10 minutes and swaps a new leaf into `/run/hive-agent-bao-identity/`, which every store login in the container presents. It reads under the leaf it holds, so it renews only while that leaf is valid |
| `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive; `swarm-logs` in the agent container, under the same certificate, on every query, as the bearer it presents at the log store; `swarm-nats-auth` and `swarm-logs-auth`, each under its own certificate, to check a presented secret | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff. `swarm-logs` reads it on every query and once more after a 401. `swarm-logs-auth` keeps granting a secret it verified for `deploy.victorialogs.authVerdictTtl` (60 s by default) without reading the path, so a replaced or deleted secret still works at the log store for up to that long | | `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive; `swarm-logs` and `swarm-metrics` in the agent container, under the same certificate, on every query, as the bearer they present at the log and metrics stores; `swarm-nats-auth` and `swarm-logs-auth`, each under its own certificate, and `swarm-metrics-agent-auth` under `swarm-metrics-auth`'s, to check a presented secret | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff. `swarm-logs` and `swarm-metrics` read it on every query and once more after a 401. `swarm-logs-auth` keeps granting a secret it verified for `deploy.victorialogs.authVerdictTtl` (60 s by default) without reading the path, so a replaced or deleted secret still works at the log store for up to that long; `swarm-metrics-agent-auth` likewise at the metrics store, for `deploy.victoriametrics.agentAuthVerdictTtl` |
| `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer | | `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer |
| `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when an operator links an external forge account in the swarm UI | `hive-agent-forge-accounts` in the agent container, under the agent's own certificate, into `<state>/forge-<label>-token` and `forge-<label>.json` | ❌ an operator's token; replaced only by deleting it and linking the label again | ✅ the agent re-fetches on a 2-minute timer | | `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when an operator links an external forge account in the swarm UI | `hive-agent-forge-accounts` in the agent container, under the agent's own certificate, into `<state>/forge-<label>-token` and `forge-<label>.json` | ❌ an operator's token; replaced only by deleting it and linking the label again | ✅ the agent re-fetches on a 2-minute timer |
| `swarm/agents/<agent>/github-token` | `swarm-controller`, when an operator links a GitHub account in the swarm UI | `hive-agent-github-token` in the agent container, under the agent's own certificate, into `<state>/github-token` | ❌ an operator's token; replaced only by deleting it and linking it again | ✅ the agent re-fetches on a 2-minute timer, and `hive-github-notify` re-reads the file on every poll | | `swarm/agents/<agent>/github-token` | `swarm-controller`, when an operator links a GitHub account in the swarm UI | `hive-agent-github-token` in the agent container, under the agent's own certificate, into `<state>/github-token` | ❌ an operator's token; replaced only by deleting it and linking it again | ✅ the agent re-fetches on a 2-minute timer, and `hive-github-notify` re-reads the file on every poll |

28
docs/swarm/services.md
View file

@ -1,3 +1,7 @@
| `deploy.victoriametrics.agentAuthRate` | Default `50r/s` per client address, for agent reads checked by `swarm-metrics-agent-auth`; over it and `agentAuthBurst`, the gateway answers 403. |
| `deploy.victoriametrics.agentAuthBurst` | Default `500` requests above `agentAuthRate` before the 403s start. |
| `deploy.victoriametrics.agentAuthVerdictTtl` | Default `60` seconds that `swarm-metrics-agent-auth` keeps granting an agent secret it verified. A replaced or deleted secret keeps working at the metrics store for up to that long. |
# Swarm-wide services # Swarm-wide services
Some things exist once per **swarm** rather than once per hive: the forge, Some things exist once per **swarm** rather than once per hive: the forge,
@ -201,11 +205,14 @@ VictoriaMetrics authenticates nothing itself; the `metrics.<swarm-domain>`
vhost does it: vhost does it:
- **The read API** (the Prometheus query, series, label and metadata - **The read API** (the Prometheus query, series, label and metadata
routes): Grafana's datasource or an operator. Grafana sends a bearer that routes): Grafana's datasource, an agent, or an operator. Grafana sends a
`swarm-bao-grafana-metrics-bearer` creates in the swarm secret store, and bearer that `swarm-bao-grafana-metrics-bearer` creates in the swarm secret
`swarm-metrics-auth` on the store's host checks it against the same store store, and `swarm-metrics-auth` on the store's host checks it against the
path. A request without a bearer needs an operator's authelia session same store path. An agent's `swarm-metrics` sends its
(group `admins`); without one it gets a 401, not a login redirect. `swarm-agent.<agent>.<secret>` bearer, which `swarm-metrics-agent-auth` on
the store's host checks against `swarm/agents/<agent>/queue`. A request
without a bearer needs an operator's authelia session (group `admins`);
without one it gets a 401, not a login redirect.
- **vmui**: an operator's authelia session, with a redirect to the login. - **vmui**: an operator's authelia session, with a redirect to the login.
The bare domain redirects to vmui. The bare domain redirects to vmui.
- **The collector's OTLP push, and its scrape of the store's own - **The collector's OTLP push, and its scrape of the store's own
@ -213,13 +220,16 @@ vhost does it:
any other session or client there, operators included. any other session or client there, operators included.
- **Every other path**, the store's write and admin API included: 403. - **Every other path**, the store's write and admin API included: 403.
`swarm-metrics-auth` reads the store with a leaf of its own, `swarm-metrics-auth` and `swarm-metrics-agent-auth` read the store with one
`deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`. leaf, `deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`.
Where the secret store runs on the same host, these default to the leaf it Where the secret store runs on the same host, these default to the leaf it
mints. On any other host you place a leaf whose CN is mints. On any other host you place a leaf whose CN is
`deploy.bao.metricsAuthCommonName`, and evaluation refuses the config until `deploy.bao.metricsAuthCommonName`, and evaluation refuses the config until
you set both. It reads the stored bearer at most once a minute, so a you set both. `swarm-metrics-auth` reads the stored bearer at most once a
replaced bearer starts working up to a minute later. minute, so a replaced bearer starts working up to a minute later.
`swarm-metrics-agent-auth` is the log store's `swarm-logs-auth` binary, with
the same grant TTL and rate limit, under `deploy.victoriametrics.agentAuth*`
(table above).
The store's container has no network, so its journal forwarder leaves through The store's container has no network, so its journal forwarder leaves through
a relay. Inside the container, `swarm-victoriametrics-egress-relay` listens on a relay. Inside the container, `swarm-victoriametrics-egress-relay` listens on

123
docs/tools/swarm-metrics-cli.md Normal file
View file

@ -0,0 +1,123 @@
# Command-Line Help for `swarm-metrics`
This document contains the help content for the `swarm-metrics` command-line program.
**Command Overview:**
* [`swarm-metrics`↴](#swarm-metrics)
* [`swarm-metrics query`↴](#swarm-metrics-query)
* [`swarm-metrics query-range`↴](#swarm-metrics-query-range)
* [`swarm-metrics series`↴](#swarm-metrics-series)
* [`swarm-metrics labels`↴](#swarm-metrics-labels)
* [`swarm-metrics label-values`↴](#swarm-metrics-label-values)
## `swarm-metrics`
query the swarm's metrics store from an agent
**Usage:** `swarm-metrics <COMMAND>`
###### **Subcommands:**
* `query` — Evaluate a `MetricsQL` (`PromQL`) expression at one instant
* `query-range` — Evaluate an expression over a window, one point per step
* `series` — List the series that match one or more selectors
* `labels` — List label names, optionally only those on matching series
* `label-values` — List one label's values; `__name__` lists every metric name
## `swarm-metrics query`
Evaluate a `MetricsQL` (`PromQL`) expression at one instant
**Usage:** `swarm-metrics query [OPTIONS] <PROMQL>`
###### **Arguments:**
* `<PROMQL>` — The expression. Quote it: `{`, `|` and `"` are the shell's too.
OTLP metric names keep their dots, so match them by `{__name__="jobq.nodes"}` rather than as a bare name.
###### **Options:**
* `--time <TIME>` — The instant: RFC 3339, a unix timestamp, or relative (`-5m`). Unset means now
## `swarm-metrics query-range`
Evaluate an expression over a window, one point per step
**Usage:** `swarm-metrics query-range [OPTIONS] --start <START> <PROMQL>`
###### **Arguments:**
* `<PROMQL>` — The expression, quoted as for `query`
###### **Options:**
* `--start <START>` — Start of the window: RFC 3339, a unix timestamp, or relative (`-1h`). Required, so a range is always bounded
* `--end <END>` — End of the window, in the same forms. Unset means now
* `--step <STEP>` — Seconds or a duration (`1m`) between points. Unset means the store's default
## `swarm-metrics series`
List the series that match one or more selectors
**Usage:** `swarm-metrics series [OPTIONS] <MATCHERS>...`
###### **Arguments:**
* `<MATCHERS>` — A series selector, such as `{"service.name"="nats"}`. Repeatable
###### **Options:**
* `--start <START>` — Start of the window: RFC 3339, a unix timestamp, or relative (`-1h`).
Unset means the store's default: the last day, from 00:00 UTC.
* `--end <END>` — End of the window, in the same forms. Unset means now
* `--limit <LIMIT>` — Return at most this many entries
## `swarm-metrics labels`
List label names, optionally only those on matching series
**Usage:** `swarm-metrics labels [OPTIONS]`
###### **Options:**
* `--match <MATCHERS>` — A series selector to narrow to. Repeatable
* `--start <START>` — Start of the window: RFC 3339, a unix timestamp, or relative (`-1h`).
Unset means the store's default: the last day, from 00:00 UTC.
* `--end <END>` — End of the window, in the same forms. Unset means now
* `--limit <LIMIT>` — Return at most this many entries
## `swarm-metrics label-values`
List one label's values; `__name__` lists every metric name
**Usage:** `swarm-metrics label-values [OPTIONS] <NAME>`
###### **Arguments:**
* `<NAME>` — The label name
###### **Options:**
* `--match <MATCHERS>` — A series selector to narrow to. Repeatable
* `--start <START>` — Start of the window: RFC 3339, a unix timestamp, or relative (`-1h`).
Unset means the store's default: the last day, from 00:00 UTC.
* `--end <END>` — End of the window, in the same forms. Unset means now
* `--limit <LIMIT>` — Return at most this many entries

11
docs/trust-boundary/security.md
View file

@ -84,12 +84,13 @@ A hive reads no **swarm service's** OIDC client secret. Every reader of
collector and the store forwarder each fetch theirs with a leaf of their own, collector and the store forwarder each fetch theirs with a leaf of their own,
and the store's granter fetches the bao UI's. and the store's granter fetches the bao UI's.
Two services read **every agent's queue secret** Three services read **every agent's queue secret**
(`swarm/agents/<agent>/queue`), each under a leaf of its own whose policy (`swarm/agents/<agent>/queue`), each under a leaf of its own whose policy
reaches that one leaf per agent and nothing else: the queue's auth callout reaches that one leaf per agent: the queue's auth callout (`swarm-nats-auth`),
(`swarm-nats-auth`) and the log store's agent check (`swarm-logs-auth`, on the the log store's agent check (`swarm-logs-auth`, on the log store's host), and
log store's host). Either one compromised can present any agent's queue the metrics store's (`swarm-metrics-agent-auth`, under the `swarm-metrics-auth`
secret, at the queue and at the log store's query route alike. leaf on the metrics store's host). Any one compromised can present any agent's
queue secret, at the queue and at both stores' read routes alike.
### Per-hive credentials live under the hive kind ### Per-hive credentials live under the hive kind

1
flake.nix
View file

@ -96,6 +96,7 @@
hive-screen-mcp hive-screen-mcp
hive-subagent-daemon hive-subagent-daemon
swarm-logs swarm-logs
swarm-metrics
assets assets
frontend frontend
reference-docs reference-docs

26
hive-c0re/src/meta.rs
View file

@ -387,6 +387,8 @@ const FORWARDED_VARS: &[&str] = &[
// same reason as the two above: it is a gateway address, and a container // same reason as the two above: it is a gateway address, and a container
// cannot derive it from anything it holds. // cannot derive it from anything it holds.
"HIVE_AGENT_LOGS_QUERY_URL", "HIVE_AGENT_LOGS_QUERY_URL",
// The metrics store's origin, for `swarm-metrics`, for the same reason.
"HIVE_AGENT_METRICS_URL",
"HIVE_FORGE_URL", "HIVE_FORGE_URL",
"HIVE_FORGE_PUBLIC_URL", "HIVE_FORGE_PUBLIC_URL",
"HIVE_MATRIX_URL", "HIVE_MATRIX_URL",
@ -433,6 +435,11 @@ const FORWARDED_VAR_OPTIONS: &[(&str, &str)] = &[
"HIVE_AGENT_LOGS_QUERY_URL", "HIVE_AGENT_LOGS_QUERY_URL",
"services.hyperhive.agent.logs.queryUrl", "services.hyperhive.agent.logs.queryUrl",
), ),
// The same for `swarm-metrics`: see `nix/agent-modules/metrics.nix`.
(
"HIVE_AGENT_METRICS_URL",
"services.hyperhive.agent.metrics.url",
),
("HIVE_FORGE_URL", "services.hyperhive.agent.forge.url"), ("HIVE_FORGE_URL", "services.hyperhive.agent.forge.url"),
("HIVE_MATRIX_URL", "services.hyperhive.agent.matrix.url"), ("HIVE_MATRIX_URL", "services.hyperhive.agent.matrix.url"),
("HYPERHIVE_SWARM_NAME", "services.hyperhive.agent.swarmName"), ("HYPERHIVE_SWARM_NAME", "services.hyperhive.agent.swarmName"),
@ -1644,6 +1651,25 @@ mod tests {
); );
} }
/// `swarm-metrics` is installed only when the option is set, and reads
/// the env var at runtime, so the metrics store's origin needs both halves.
#[test]
fn the_metrics_url_is_forwarded_as_both_env_and_option() {
assert!(FORWARDED_VARS.contains(&"HIVE_AGENT_METRICS_URL"));
let mut out = String::new();
push_forwarded_var_options(
&mut out,
&[(
"HIVE_AGENT_METRICS_URL",
"https://metrics.t.local".to_string(),
)],
);
assert_eq!(
out,
" services.hyperhive.agent.metrics.url = \"https://metrics.t.local\";\n"
);
}
#[test] #[test]
fn require_service_urls_accepts_a_rendered_forge_url() { fn require_service_urls_accepts_a_rendered_forge_url() {
require_service_urls(&[ require_service_urls(&[

1
nix/agent-modules/default.nix
View file

@ -48,6 +48,7 @@ in
./matrix.nix ./matrix.nix
./mcp.nix ./mcp.nix
./memory-cap.nix ./memory-cap.nix
./metrics.nix
./network.nix ./network.nix
./otel.nix ./otel.nix
./otel-removed.nix ./otel-removed.nix

66
nix/agent-modules/metrics.nix Normal file
View file

@ -0,0 +1,66 @@
# `swarm-metrics` on this agent's PATH, configured for this swarm's metrics
# store. The sibling of ./logs.nix, in the same shape for the same reasons:
# the binary defaults nothing (see `swarm-metrics/README.md`), so it is
# installed wrapped, and it is gated on the secret store's address as well as
# on the store URL, because the identity it presents is the agent's own queue
# secret read from that store.
{
lib,
pkgs,
config,
...
}:
let
cfg = config.services.hyperhive.agent.metrics;
baoCfg = config.services.hyperhive.agent.bao;
configured = cfg.url != null && baoCfg.addr != null;
# The coordinates ./logs.nix gives `swarm-logs`. Paths and a name only: a
# value in `--set` lands in the world-readable nix store.
wrapperEnv = {
HIVE_AGENT_METRICS_URL = cfg.url;
HIVE_AGENT_NAME = config.services.hyperhive.agent.user.name;
BAO_ADDR = baoCfg.addr;
BAO_CLIENT_CERT = baoCfg.clientCertFile;
BAO_CLIENT_KEY = baoCfg.clientKeyFile;
BAO_CACERT = "/run/credentials/hive-agent.service/hive-agent-bao-server-ca";
};
swarmMetricsConfigured = pkgs.symlinkJoin {
name = "swarm-metrics-configured";
paths = [ config.services.hyperhive.agent.packages.swarm-metrics ];
nativeBuildInputs = [ pkgs.makeWrapper ];
postBuild = ''
wrapProgram $out/bin/swarm-metrics ${
lib.concatStringsSep " " (
lib.mapAttrsToList (name: value: "--set ${name} ${lib.escapeShellArg value}") wrapperEnv
)
}
'';
};
in
{
options.services.hyperhive.agent.metrics = {
url = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "https://metrics.example.com";
description = ''
The swarm metrics store's gateway origin, as this container reaches
it. `swarm-metrics` appends the read API's paths (`/api/v1/query`,
...) to it.
Set by the generated meta flake from the host's own
{option}`services.hyperhive.swarm.victoriametrics.domain`.
`null` means this hive has not been given the metrics store's address
for its agents, and `swarm-metrics` is then not installed at all.
Not defaulted: a guessed address resolves cleanly to the wrong place.
'';
};
};
config = lib.mkIf configured {
environment.systemPackages = [ swarmMetricsConfigured ];
};
}

15
nix/checks.nix
View file

@ -478,6 +478,21 @@ in
touch "$out" touch "$out"
''; '';
# `swarm-metrics` CLI reference freshness check, the same shape as
# `swarm-logs-docs` above.
swarm-metrics-docs =
pkgs.runCommand "swarm-metrics-docs-fresh" { nativeBuildInputs = [ pkgs.diffutils ]; }
''
${self.packages.${system}.swarm-metrics}/bin/swarm-metrics markdown-docs > generated.md
if ! diff -u ${../docs/tools/swarm-metrics-cli.md} generated.md; then
echo "" >&2
echo "ERROR: docs/tools/swarm-metrics-cli.md is out of date — regenerate it:" >&2
echo " nix build .#swarm-metrics && ./result/bin/swarm-metrics markdown-docs > docs/tools/swarm-metrics-cli.md" >&2
exit 1
fi
touch "$out"
'';
# `hive-forge` CLI reference freshness check — same shape as # `hive-forge` CLI reference freshness check — same shape as
# `hivectl-docs` above, `hive-forge markdown-docs` (clap-markdown over # `hivectl-docs` above, `hive-forge markdown-docs` (clap-markdown over
# its own command tree) instead. Reuses `packages.<system>.hive-forge` # its own command tree) instead. Reuses `packages.<system>.hive-forge`

4
nix/host-modules/glue-bao-tls.nix
View file

@ -285,8 +285,8 @@ in
[ -s ${pkiDir}/logs-auth.pem ] || ${signLeaf} ${pkiDir} logs-auth \ [ -s ${pkiDir}/logs-auth.pem ] || ${signLeaf} ${pkiDir} logs-auth \
${lib.escapeShellArg deployCfg.bao.logsAuthCommonName} "" clientAuth ${lib.escapeShellArg deployCfg.bao.logsAuthCommonName} "" clientAuth
# The metrics store's Grafana auth responder, which reads Grafana's # The metrics store's auth responders, which read Grafana's metrics
# metrics bearer. Minted here for the matrix-ctl leaf's reason: the # bearer and agents' queue secrets. Minted here for the matrix-ctl leaf's reason: the
# metrics store is a swarm singleton, so elsewhere this is the file an # metrics store is a swarm singleton, so elsewhere this is the file an
# operator copies. # operator copies.
[ -s ${pkiDir}/metrics-auth.pem ] || ${signLeaf} ${pkiDir} metrics-auth \ [ -s ${pkiDir}/metrics-auth.pem ] || ${signLeaf} ${pkiDir} metrics-auth \

4
nix/host-modules/glue-metrics-auth-bao-identity.nix
View file

@ -1,5 +1,5 @@
# Glue: point the metrics store's Grafana auth responder at the bao leaf minted # Glue: point the metrics store's auth responders (Grafana's and the agents')
# for it. # at the bao leaf minted for them.
# #
# ONE PAIRING PER FILE — the swarm-metrics-auth principal ← bao, and nothing # ONE PAIRING PER FILE — the swarm-metrics-auth principal ← bao, and nothing
# else. Deleting this leaves a metrics store with no responder identity unless # else. Deleting this leaves a metrics store with no responder identity unless

3
nix/host-modules/hive-c0re/environment.nix
View file

@ -287,6 +287,9 @@ in
# `swarm-victorialogs.nix`'s location for the whole reasoning. # `swarm-victorialogs.nix`'s location for the whole reasoning.
lib.optionalAttrs haveBaoClientIdentity { lib.optionalAttrs haveBaoClientIdentity {
HIVE_AGENT_LOGS_QUERY_URL = "https://${config.services.hyperhive.swarm.victorialogs.domain}/select/logsql/query"; HIVE_AGENT_LOGS_QUERY_URL = "https://${config.services.hyperhive.swarm.victorialogs.domain}/select/logsql/query";
# The metrics store's origin, for `swarm-metrics`; same credential and
# gate. Its read locations answer a bare 401, never a login redirect.
HIVE_AGENT_METRICS_URL = "https://${config.services.hyperhive.swarm.victoriametrics.domain}";
} }
// //
# Where the swarm's secret store is, and the identity this hive presents to # Where the swarm's secret store is, and the identity this hive presents to

16
nix/host-modules/swarm-bao.nix
View file

@ -1060,15 +1060,18 @@ let
} }
]; ];
# The metrics store's responder, which checks the bearer Grafana presents # The metrics store's responders under one leaf: the agent one checks the
# against the one stored for it; and the host unit that fetches the store # secret an agent presents against the one stored for it (the same `+` leaf
# container's journal forwarder secret under the same leaf. Nothing else. # as the log store's), the other checks the bearer Grafana presents; and the
# host unit that fetches the store container's journal forwarder secret.
# Nothing else.
metricsAuthReaders = [ metricsAuthReaders = [
{ {
name = "swarm-metrics-auth"; name = "swarm-metrics-auth";
cn = baoDeploy.metricsAuthCommonName; cn = baoDeploy.metricsAuthCommonName;
policyText = policyText =
readStanza "${credentialMountPath}/data/${grafanaMetricsBearerLeaf}" readStanza "${credentialMountPath}/data/swarm/agents/+/queue"
+ readStanza "${credentialMountPath}/data/${grafanaMetricsBearerLeaf}"
# The metrics store container's journal forwarder, fetched on its host. # The metrics store container's journal forwarder, fetched on its host.
+ readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.victoriametrics.otel.clientId}/oidc/client"; + readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.victoriametrics.otel.clientId}/oidc/client";
} }
@ -1977,8 +1980,9 @@ in
default = "swarm-metrics-auth"; default = "swarm-metrics-auth";
description = '' description = ''
Subject the store's `swarm-metrics-auth` cert-auth role accepts: the Subject the store's `swarm-metrics-auth` cert-auth role accepts: the
identity the metrics store's Grafana auth responder presents to read identity the metrics store's auth responders present to read agent
the bearer Grafana's datasource sends, queue credentials, `swarm/agents/<agent>/queue`, and the bearer
Grafana's datasource sends,
`swarm/services/<grafana client id>/metrics/bearer`, and the one `swarm/services/<grafana client id>/metrics/bearer`, and the one
`swarm-victoriametrics-forwarder-oidc` presents to read `swarm-victoriametrics-forwarder-oidc` presents to read
`swarm/services/<metrics store forwarder client id>/oidc/client`. `swarm/services/<metrics store forwarder client id>/oidc/client`.

135
nix/host-modules/swarm-victoriametrics.nix
View file

@ -38,9 +38,9 @@ let
nginxGid = config.ids.gids.nginx; nginxGid = config.ids.gids.nginx;
# The store's read API, which the vhost forwards to a caller holding # The store's read API, which the vhost forwards to a caller holding
# Grafana's bearer or an operator's session. Each location's `proxy_pass` # Grafana's bearer, an agent's own bearer, or an operator's session. Each
# carries the path, so the store receives the path nginx matched, not the # location's `proxy_pass` carries the path, so the store receives the path
# caller's unnormalised spelling. # nginx matched, not the caller's unnormalised spelling.
# #
# Grafana's Prometheus datasource reads the query, series, label, metadata, # Grafana's Prometheus datasource reads the query, series, label, metadata,
# buildinfo, exemplar, rules and alerts routes; the store answers the last # buildinfo, exemplar, rules and alerts routes; the store answers the last
@ -128,6 +128,14 @@ let
# same leaf as `grafanaMetricsBearerLeaf` in ./swarm-bao.nix. # same leaf as `grafanaMetricsBearerLeaf` in ./swarm-bao.nix.
grafanaBearerStorePath = "swarm/services/${hyperhiveCfg.swarm.grafana.oidc.clientId}/metrics/bearer"; grafanaBearerStorePath = "swarm/services/${hyperhiveCfg.swarm.grafana.oidc.clientId}/metrics/bearer";
# The agents' check on the read API: a `swarm-logs-auth` binary, logging in
# under this host's `swarm-metrics-auth` leaf, whose grant covers every
# agent's queue secret. The log store's host runs `swarm-metrics-auth` under
# its `swarm-logs-auth` leaf the same way.
agentAuthUnit = "swarm-metrics-agent-auth";
agentAuthSocket = "/run/${agentAuthUnit}/auth.sock";
agentAuthCredential = id: "/run/credentials/${agentAuthUnit}.service/${id}";
# ── the container's journal forwarder ─────────────────────────────── # ── the container's journal forwarder ───────────────────────────────
# #
# With no interface but loopback, the forwarder reaches nothing by name. A # With no interface but loopback, the forwarder reaches nothing by name. A
@ -242,7 +250,8 @@ in
example = "/var/lib/swarm-bao-pki/metrics-auth.pem"; example = "/var/lib/swarm-bao-pki/metrics-auth.pem";
description = '' description = ''
Client certificate `swarm-metrics-auth` presents to the swarm's secret Client certificate `swarm-metrics-auth` presents to the swarm's secret
store to read Grafana's metrics bearer. Its subject must be store to read Grafana's metrics bearer, and `swarm-metrics-agent-auth`
to read agents' queue secrets. Its subject must be
{option}`services.hyperhive.deploy.bao.metricsAuthCommonName`. {option}`services.hyperhive.deploy.bao.metricsAuthCommonName`.
Required when `deploy.victoriametrics.enable` is set; unset fails at Required when `deploy.victoriametrics.enable` is set; unset fails at
@ -264,6 +273,47 @@ in
''; '';
}; };
options.services.hyperhive.deploy.victoriametrics.agentAuthVerdictTtl = lib.mkOption {
type = lib.types.ints.unsigned;
default = 60;
description = ''
Seconds `swarm-metrics-agent-auth` grants an agent bearer it has
verified again without reading the secret store. A denial is never
remembered.
This is also how long a re-minted or deleted queue secret keeps working
at the metrics store's read API after its last successful check. `0`
checks the store on every query.
'';
};
options.services.hyperhive.deploy.victoriametrics.agentAuthRate = lib.mkOption {
type = lib.types.str;
default = "50r/s";
example = "30r/m";
description = ''
nginx `limit_req` rate, per client address, of read API requests
carrying a `swarm-agent.` bearer, which nginx checks with
`swarm-metrics-agent-auth`. Each such request that the responder has
not granted within
{option}`services.hyperhive.deploy.victoriametrics.agentAuthVerdictTtl`
costs a secret store login and read, whether the bearer is right or not.
Keyed by client address, so every agent on one host shares one budget.
Requests over the rate and its burst are answered 403.
'';
};
options.services.hyperhive.deploy.victoriametrics.agentAuthBurst = lib.mkOption {
type = lib.types.ints.positive;
default = 500;
description = ''
Requests a client address may make above
{option}`services.hyperhive.deploy.victoriametrics.agentAuthRate` before
nginx answers 403, served without delay.
'';
};
config = lib.mkIf deployCfg.victoriametrics.enable { config = lib.mkIf deployCfg.victoriametrics.enable {
assertions = [ assertions = [
{ {
@ -287,7 +337,8 @@ in
services.hyperhive.deploy.victoriametrics.authBaoClientCertFile and services.hyperhive.deploy.victoriametrics.authBaoClientCertFile and
authBaoClientKeyFile must both be set. authBaoClientKeyFile must both be set.
swarm-metrics-auth reads Grafana's metrics bearer from the swarm's swarm-metrics-auth reads Grafana's metrics bearer, and
swarm-metrics-agent-auth agents' queue secrets, from the swarm's
secret store with this certificate. They default to the leaf the secret store with this certificate. They default to the leaf the
store mints where services.hyperhive.deploy.bao.enable is set on store mints where services.hyperhive.deploy.bao.enable is set on
this host; elsewhere, issue a leaf whose CN is this host; elsewhere, issue a leaf whose CN is
@ -320,14 +371,18 @@ in
# Which check a read location's `auth_request` gets. `auth_request` takes a # Which check a read location's `auth_request` gets. `auth_request` takes a
# fixed URI, so it names `/__metrics_read_auth`, which rewrites to the # fixed URI, so it names `/__metrics_read_auth`, which rewrites to the
# target this map picks: a bearer goes to Grafana's check, anything else # target this map picks: an agent's `swarm-agent.` bearer goes to the
# (an operator's session cookie) to authelia. After the forwarder's egress # agents' check, any other bearer to Grafana's, anything else (an
# operator's session cookie) to authelia. A map tries its regexes in
# order, so the agent arm comes first. After the forwarder's egress
# listener, `egressServer`. # listener, `egressServer`.
services.nginx.appendHttpConfig = egressServer + '' services.nginx.appendHttpConfig = egressServer + ''
map $http_authorization $swarm_metrics_read_auth_target { map $http_authorization $swarm_metrics_read_auth_target {
"~^Bearer swarm-agent\." /__metrics_agent_auth;
"~^Bearer " /__metrics_grafana_auth; "~^Bearer " /__metrics_grafana_auth;
default /__metrics_authelia; default /__metrics_authelia;
} }
limit_req_zone $binary_remote_addr zone=swarm_metrics_agent_auth:1m rate=${deployCfg.victoriametrics.agentAuthRate};
''; '';
# `forceSSL`: authelia answers a plain-http auth subrequest # `forceSSL`: authelia answers a plain-http auth subrequest
@ -400,6 +455,22 @@ in
''; '';
}; };
# An agent's bearer, checked by `swarm-metrics-agent-auth`. Any
# well-formed agent bearer the responder has not recently granted
# costs a store login and read, so the rate per client address is
# capped here. 403 when over: `auth_request` passes only 401 and 403
# through, and `swarm-metrics` retries a 401.
"= /__metrics_agent_auth" = {
proxyPass = "http://unix:${agentAuthSocket}:/";
extraConfig = ''
internal;
limit_req zone=swarm_metrics_agent_auth burst=${toString deployCfg.victoriametrics.agentAuthBurst} nodelay;
limit_req_status 403;
proxy_pass_request_body off;
proxy_set_header Content-Length "";
'';
};
# The authelia subrequest. For the collector's two locations above it # The authelia subrequest. For the collector's two locations above it
# admits only the collector's client; for the read and vmui locations # admits only the collector's client; for the read and vmui locations
# it admits an operator's session (`metricsStoreRules` in # it admits an operator's session (`metricsStoreRules` in
@ -506,6 +577,56 @@ in
}; };
}; };
# The agents' check on the read API, see `agentAuthUnit`.
users.users.${agentAuthUnit} = {
isSystemUser = true;
group = agentAuthUnit;
description = "swarm metrics store auth_request responder for agent credentials";
};
users.groups.${agentAuthUnit} = { };
systemd.services.${agentAuthUnit} = {
description = "swarm metrics store auth_request responder for agent credentials";
wantedBy = [ "multi-user.target" ];
after = [
"network.target"
"swarm-bao-pki.service"
];
serviceConfig = {
ExecStart = lib.concatStringsSep " " [
"${deployCfg.victorialogs.authPackage}/bin/swarm-logs-auth"
"--socket ${agentAuthSocket}"
"--store-cert-role swarm-metrics-auth"
"--verdict-ttl ${toString deployCfg.victoriametrics.agentAuthVerdictTtl}"
];
LoadCredential = lib.mapAttrsToList (id: path: "${id}:${path}") metricsAuthStoreFiles;
# ⚠️ A static user, never `DynamicUser`: see the `swarm-metrics-auth` unit above.
User = agentAuthUnit;
Group = agentAuthUnit;
RuntimeDirectory = agentAuthUnit;
RuntimeDirectoryMode = "0751";
RuntimeDirectoryPreserve = "yes";
Restart = "on-failure";
RestartSec = "5s";
SyslogIdentifier = agentAuthUnit;
ProtectSystem = "strict";
ProtectHome = true;
PrivateTmp = "disconnected";
PrivateDevices = true;
NoNewPrivileges = true;
RestrictSUIDSGID = true;
RemoveIPC = true;
};
environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
BAO_CLIENT_CERT = agentAuthCredential "bao-client.pem";
BAO_CLIENT_KEY = agentAuthCredential "bao-client-key.pem";
}
// lib.optionalAttrs (metricsAuthStoreFiles ? "bao-ca.pem") {
BAO_CACERT = agentAuthCredential "bao-ca.pem";
};
};
# nginx runs as its own user under `ProtectSystem=strict`, so it can only # nginx runs as its own user under `ProtectSystem=strict`, so it can only
# create the socket where it may write. It removes the socket on a clean # create the socket where it may write. It removes the socket on a clean
# exit only; a stale one after a crash fails every later `bind`, and with # exit only; a stale one after a crash fails every later `bind`, and with

41
nix/module-eval/agent-queue-bao.nix
View file

@ -41,19 +41,23 @@ let
agentNoBao = shared.agentBare; agentNoBao = shared.agentBare;
# `swarm-logs` authenticates with the agent's queue secret from the store, so # `swarm-logs` and `swarm-metrics` authenticate with the agent's queue secret
# the store address gates it beside the query URL. # from the store, so the store address gates each beside its store URL.
logsUrl = "https://logs.t.local/select/logsql/query"; logsUrl = "https://logs.t.local/select/logsql/query";
metricsUrl = "https://metrics.t.local";
agentLogs = agentWith { agentLogs = agentWith {
services.hyperhive.agent.bao.addr = "https://bao.t.local:8200"; services.hyperhive.agent.bao.addr = "https://bao.t.local:8200";
services.hyperhive.agent.logs.queryUrl = logsUrl; services.hyperhive.agent.logs.queryUrl = logsUrl;
services.hyperhive.agent.metrics.url = metricsUrl;
}; };
agentLogsNoBao = agentWith { services.hyperhive.agent.logs.queryUrl = logsUrl; }; agentLogsNoBao = agentWith {
swarmLogsIn = services.hyperhive.agent.logs.queryUrl = logsUrl;
machine: services.hyperhive.agent.metrics.url = metricsUrl;
lib.findFirst ( };
p: (p.name or "") == "swarm-logs-configured" configuredIn =
) null machine.environment.systemPackages; name: machine: lib.findFirst (p: (p.name or "") == name) null machine.environment.systemPackages;
swarmLogsIn = configuredIn "swarm-logs-configured";
swarmMetricsIn = configuredIn "swarm-metrics-configured";
agentBaoIdentity = machine: machine.systemd.services.hive-agent-bao-identity; agentBaoIdentity = machine: machine.systemd.services.hive-agent-bao-identity;
cases = [ cases = [
@ -80,6 +84,27 @@ let
name = "an agent with a log store but no secret store gets no swarm-logs"; name = "an agent with a log store but no secret store gets no swarm-logs";
ok = swarmLogsIn agentLogsNoBao == null && swarmLogsIn agentLogs != null; ok = swarmLogsIn agentLogsNoBao == null && swarmLogsIn agentLogs != null;
} }
{
# The same store identity as `swarm-logs`, and the store's origin.
name = "swarm-metrics is wrapped with the agent's own store identity";
ok =
let
w = (swarmMetricsIn agentLogs).buildCommand;
a = agentLogs.services.hyperhive.agent;
set = k: v: lib.hasInfix "--set ${k} ${lib.escapeShellArg v}" w;
in
set "HIVE_AGENT_METRICS_URL" metricsUrl
&& set "HIVE_AGENT_NAME" a.user.name
&& set "BAO_ADDR" "https://bao.t.local:8200"
&& set "BAO_CLIENT_CERT" a.bao.clientCertFile
&& set "BAO_CLIENT_KEY" a.bao.clientKeyFile
&& set "BAO_CACERT" "/run/credentials/hive-agent.service/hive-agent-bao-server-ca";
}
{
# A metrics URL with nothing to authenticate as installs nothing.
name = "an agent with a metrics store but no secret store gets no swarm-metrics";
ok = swarmMetricsIn agentLogsNoBao == null && swarmMetricsIn agentLogs != null;
}
{ {
# Both ids or neither: the secret authenticates nobody without the id it # Both ids or neither: the secret authenticates nobody without the id it
# belongs to, and `swarm-logs` reads both. # belongs to, and `swarm-logs` reads both.

14
nix/module-eval/bao-grants.nix
View file

@ -788,16 +788,20 @@ let
lib.hasInfix "/logs-auth.pem ]" s && lib.hasInfix "swarm-logs-auth \"\" clientAuth" s; lib.hasInfix "/logs-auth.pem ]" s && lib.hasInfix "swarm-logs-auth \"\" clientAuth" s;
} }
{ {
# Read-only on the one leaf Grafana's host creates and on the store # Read-only on every agent's queue secret, the one leaf Grafana's host
# container forwarder's client secret, and nothing else: this principal # creates and the store container forwarder's client secret, and nothing
# checks a presented bearer, and its host fetches one secret for the # else: this principal checks presented bearers, and its host fetches
# container. # one secret for the container.
name = "the metrics store responder's grant is Grafana's metrics bearer and its container forwarder's client secret, read-only, and nothing else"; name = "the metrics store responder's grant is agents' queue secrets, Grafana's metrics bearer and its container forwarder's client secret, read-only, and nothing else";
ok = ok =
let let
s = baoGrantHere.systemd.services.swarm-bao-metrics-auth-policy.script; s = baoGrantHere.systemd.services.swarm-bao-metrics-auth-policy.script;
in in
grantsIn s == [ grantsIn s == [
{
path = "secret/data/swarm/agents/+/queue";
caps = [ "read" ];
}
{ {
path = "secret/data/swarm/services/swarm-grafana/metrics/bearer"; path = "secret/data/swarm/services/swarm-grafana/metrics/bearer";
caps = [ "read" ]; caps = [ "read" ];

31
nix/module-eval/core-toggle.nix
View file

@ -672,15 +672,21 @@ let
{ {
# The read API's check, link by link: every read location names the # The read API's check, link by link: every read location names the
# switch with no login redirect (a machine reader would get login HTML # switch with no login redirect (a machine reader would get login HTML
# as a 200), the switch rewrites to the map's pick, a bearer reaches # as a 200), the switch rewrites to the map's pick, an agent bearer
# the responder's socket, which the unit binds, and anything else goes # reaches the agents' responder (listed before the catch-all bearer arm,
# to authelia. vmui is a browser's, so it redirects to the login. # since a map takes the first matching regex), any other bearer reaches
name = "the metrics read API sends a bearer to swarm-metrics-auth and a session to authelia"; # Grafana's, each socket bound by its unit, and anything else goes to
# authelia. vmui is a browser's, so it redirects to the login.
name = "the metrics read API sends an agent bearer to swarm-metrics-agent-auth, another bearer to swarm-metrics-auth and a session to authelia";
ok = ok =
let let
l = allLocal.services.nginx.virtualHosts."metrics.t.local".locations; l = allLocal.services.nginx.virtualHosts."metrics.t.local".locations;
http = allLocal.services.nginx.appendHttpConfig; http = allLocal.services.nginx.appendHttpConfig;
unit = allLocal.systemd.services.swarm-metrics-auth; unit = allLocal.systemd.services.swarm-metrics-auth;
agentUnit = allLocal.systemd.services.swarm-metrics-agent-auth;
agentArm = "\"~^Bearer swarm-agent\\.\" /__metrics_agent_auth;";
bearerArm = "\"~^Bearer \" /__metrics_grafana_auth;";
v = allLocal.services.hyperhive.deploy.victoriametrics;
reads = lib.filterAttrs (n: _: lib.hasPrefix "= /api/v1/" n || lib.hasPrefix "^~ /api/v1/" n) l; reads = lib.filterAttrs (n: _: lib.hasPrefix "= /api/v1/" n || lib.hasPrefix "^~ /api/v1/" n) l;
in in
reads ? "= /api/v1/query_range" reads ? "= /api/v1/query_range"
@ -690,7 +696,9 @@ let
&& !(lib.hasInfix "error_page" r.extraConfig) && !(lib.hasInfix "error_page" r.extraConfig)
) (lib.attrValues reads) ) (lib.attrValues reads)
&& lib.hasInfix "map \$http_authorization \$swarm_metrics_read_auth_target" http && lib.hasInfix "map \$http_authorization \$swarm_metrics_read_auth_target" http
&& lib.hasInfix "\"~^Bearer \" /__metrics_grafana_auth;" http && lib.hasInfix agentArm http
&& lib.hasInfix bearerArm http
&& !(lib.hasInfix bearerArm (lib.head (lib.splitString agentArm http)))
&& lib.hasInfix "default /__metrics_authelia;" http && lib.hasInfix "default /__metrics_authelia;" http
&& &&
lib.hasInfix "rewrite ^ \$swarm_metrics_read_auth_target last;" lib.hasInfix "rewrite ^ \$swarm_metrics_read_auth_target last;"
@ -698,6 +706,18 @@ let
&& lib.hasInfix "internal;" l."= /__metrics_grafana_auth".extraConfig && lib.hasInfix "internal;" l."= /__metrics_grafana_auth".extraConfig
&& l."= /__metrics_grafana_auth".proxyPass == "http://unix:/run/swarm-metrics-auth/auth.sock:/" && l."= /__metrics_grafana_auth".proxyPass == "http://unix:/run/swarm-metrics-auth/auth.sock:/"
&& lib.hasInfix "--socket /run/swarm-metrics-auth/auth.sock" unit.serviceConfig.ExecStart && lib.hasInfix "--socket /run/swarm-metrics-auth/auth.sock" unit.serviceConfig.ExecStart
&& lib.hasInfix "internal;" l."= /__metrics_agent_auth".extraConfig
&& l."= /__metrics_agent_auth".proxyPass == "http://unix:/run/swarm-metrics-agent-auth/auth.sock:/"
&& lib.hasInfix "--socket /run/swarm-metrics-agent-auth/auth.sock" agentUnit.serviceConfig.ExecStart
&& lib.hasInfix "/bin/swarm-logs-auth " agentUnit.serviceConfig.ExecStart
&& lib.hasInfix "--store-cert-role swarm-metrics-auth" agentUnit.serviceConfig.ExecStart
&& lib.hasInfix "--verdict-ttl ${toString v.agentAuthVerdictTtl}" agentUnit.serviceConfig.ExecStart
&& agentUnit.serviceConfig.LoadCredential == unit.serviceConfig.LoadCredential
&& lib.hasInfix "limit_req_zone $binary_remote_addr zone=swarm_metrics_agent_auth:1m rate=${v.agentAuthRate};" http
&&
lib.hasInfix "limit_req zone=swarm_metrics_agent_auth burst=${toString v.agentAuthBurst} nodelay;"
l."= /__metrics_agent_auth".extraConfig
&& !(lib.hasInfix "limit_req" l."= /__metrics_grafana_auth".extraConfig)
&& lib.hasInfix "auth_request /__metrics_authelia;" l."^~ /vmui/".extraConfig && lib.hasInfix "auth_request /__metrics_authelia;" l."^~ /vmui/".extraConfig
&& lib.hasInfix "error_page 401 =302" l."^~ /vmui/".extraConfig && lib.hasInfix "error_page 401 =302" l."^~ /vmui/".extraConfig
&& l."= /__metrics_authelia".proxyPass == "https://auth.t.local/api/authz/auth-request"; && l."= /__metrics_authelia".proxyPass == "https://auth.t.local/api/authz/auth-request";
@ -884,6 +904,7 @@ let
owners ? swarm-logs-auth owners ? swarm-logs-auth
&& owners ? swarm-logs-grafana-auth && owners ? swarm-logs-grafana-auth
&& owners ? swarm-metrics-auth && owners ? swarm-metrics-auth
&& owners ? swarm-metrics-agent-auth
&& !(lib.any (u: hidden u.serviceConfig) (lib.attrValues owners)); && !(lib.any (u: hidden u.serviceConfig) (lib.attrValues owners));
} }
{ {

2
nix/packages/default.nix
View file

@ -43,6 +43,8 @@ let
# which is exactly what this list's per-bin packages are for — see # which is exactly what this list's per-bin packages are for — see
# `nix/agent-modules/logs.nix` for what puts it on PATH there. # `nix/agent-modules/logs.nix` for what puts it on PATH there.
swarm-logs = "hyperhive agent-facing swarm log store query CLI"; swarm-logs = "hyperhive agent-facing swarm log store query CLI";
# The same for the metrics store; `nix/agent-modules/metrics.nix` puts it on PATH.
swarm-metrics = "hyperhive agent-facing swarm metrics store query CLI";
hive-screen-mcp = "hyperhive screen MCP bridge (screenshot + input for GUI agents)"; hive-screen-mcp = "hyperhive screen MCP bridge (screenshot + input for GUI agents)";
hive-forge = "hyperhive Forgejo CLI"; hive-forge = "hyperhive Forgejo CLI";
hive-forge-notify = "hyperhive per-agent Forgejo notification poller daemon"; hive-forge-notify = "hyperhive per-agent Forgejo notification poller daemon";

20
swarm-logs-auth/src/main.rs
View file

@ -1,14 +1,16 @@
//! The log store gateway's `auth_request` responder for agents. //! The log and metrics stores' gateway `auth_request` responder for agents.
//! //!
//! nginx sends it every `/select/logsql/` request whose `Authorization` is //! nginx sends it every `/select/logsql/` request (`swarm-victorialogs.nix`),
//! `Bearer swarm-agent.<agent>.<secret>` (`swarm-victorialogs.nix`). It answers //! and every metrics read API request (`swarm-victoriametrics.nix`, as
//! 200 when the secret is the one stored at `swarm/agents/<agent>/queue`, and //! `swarm-metrics-agent-auth`), whose `Authorization` is
//! 401 otherwise, store failures included: `auth_request` turns any other //! `Bearer swarm-agent.<agent>.<secret>`. It answers 200 when the secret is
//! status into a 500 for the caller. //! the one stored at `swarm/agents/<agent>/queue`, and 401 otherwise, store
//! failures included: `auth_request` turns any other status into a 500 for
//! the caller.
//! //!
//! It reads the store under its own certificate, whose policy reaches every //! It logs in to the store as the cert-auth role `--store-cert-role` names,
//! agent's `queue` secret and nothing else. It serves a unix socket whose //! whose policy reaches every agent's `queue` secret. It serves a unix socket
//! directory only the gateway's nginx may enter. //! whose directory only the gateway's nginx may enter.
//! //!
//! A grant is remembered for `--verdict-ttl` seconds ([`verdicts`]), so an //! A grant is remembered for `--verdict-ttl` seconds ([`verdicts`]), so an
//! agent's queries within that window cost no store lookup, and a secret that //! agent's queries within that window cost no store lookup, and a secret that

27
swarm-metrics/Cargo.toml Normal file
View file

@ -0,0 +1,27 @@
[package]
name = "swarm-metrics"
version.workspace = true
readme = "README.md"
edition.workspace = true
[[bin]]
name = "swarm-metrics"
path = "src/main.rs"
[dependencies]
anyhow.workspace = true
clap.workspace = true
clap-markdown = "0.1"
# The agent-token spelling the gateway's responder parses, and the rustls
# provider the store client needs.
swarm-queue-client.workspace = true
# The read of this agent's own queue secret.
swarm-secret-client.workspace = true
# Only to drive that one async read; the query itself is blocking.
tokio.workspace = true
# Blocking, not async: this binary makes one request (two on a 401) and
# exits, so a reactor would exist only to be torn down.
reqwest = { workspace = true, features = ["blocking"] }
[lints]
workspace = true

60
swarm-metrics/README.md Normal file
View file

@ -0,0 +1,60 @@
# swarm-metrics
An agent's CLI for the swarm metrics store (VictoriaMetrics), the sibling of
`swarm-logs`. Each verb is one read against the store's Prometheus-compatible
API through the gateway, and prints the store's JSON answer unchanged, for
`jq`.
| verb | store endpoint | for |
| -------------- | ----------------------------- | --------------------------------------- |
| `query` | `/api/v1/query` | a MetricsQL expression at one instant |
| `query-range` | `/api/v1/query_range` | the same over a window; needs `--start` |
| `series` | `/api/v1/series` | which series match a selector |
| `labels` | `/api/v1/labels` | label names |
| `label-values` | `/api/v1/label/<name>/values` | one label's values (`__name__`: names) |
Read-only: nothing here writes, and the gateway serves no write path to this
caller.
## ⚠️ No scoping
The gateway forwards the request unmodified, so any authenticated caller reads
the whole swarm's metrics, every hive's.
## How it authenticates
As the agent itself, exactly as `swarm-logs` does: on each request it reads
the agent's own queue secret, `swarm/agents/<agent>/queue`, from the swarm
secret store with the agent's store certificate, and sends
`Authorization: Bearer swarm-agent.<agent>.<secret>`. A 401 makes it read the
secret again and retry once.
On the metrics store's host, nginx hands a bearer of that shape to
`swarm-metrics-agent-auth`, a `swarm-logs-auth` instance logged in to the
store under that host's `swarm-metrics-auth` leaf. It grants a secret it has
verified for 60 s by default (`deploy.victoriametrics.agentAuthVerdictTtl`).
A 403 is the gateway's per-address rate limit on these checks, and
`swarm-metrics` doesn't retry it.
🩸 The secret is never logged or rendered into an error.
## Configuration
Supplied by `nix/agent-modules/metrics.nix`, which wraps the binary.
| variable | what |
| ------------------------ | ------------------------------------------------ |
| `HIVE_AGENT_METRICS_URL` | the metrics store's gateway origin |
| `HIVE_AGENT_NAME` | this agent's name; the store path and role |
| `BAO_ADDR` | the secret store's address |
| `BAO_CLIENT_CERT` | path of the agent's live store certificate |
| `BAO_CLIENT_KEY` | path of its key |
| `BAO_CACERT` | path of the store's CA; missing or empty is fine |
With no `HIVE_AGENT_METRICS_URL` the binary says there is no metrics store;
with the URL and no store address it reports a deployment bug by name.
## Output
The store's response body, unmodified, newline-terminated. A non-2xx exits
non-zero with its status and body on stderr, never as an empty result.

149
swarm-metrics/src/auth.rs Normal file
View file

@ -0,0 +1,149 @@
//! The credential this CLI presents to the metrics store's gateway: the
//! agent's own queue secret, read from the swarm secret store with the agent's
//! store certificate.
//!
//! The same read and the same bearer as `swarm-logs`' `auth` module: path
//! `swarm/agents/<agent>/queue`, the agent's cert-auth role, and
//! `Bearer swarm-agent.<agent>.<secret>` spelled by
//! `swarm_queue_client::agent_token`. On the metrics store's host a
//! `swarm-logs-auth` instance checks it against the store.
//!
//! 🩸 The secret is never logged and never rendered into an error.
use anyhow::{Context as _, Result, bail};
use swarm_secret_client::client::{DEFAULT_CERT_MOUNT, Settings};
use swarm_secret_client::queue::{self, AgentCredential};
use swarm_secret_client::{SecretStore, policy};
/// The metrics store's gateway origin; the API paths are appended to it.
const ENV_URL: &str = "HIVE_AGENT_METRICS_URL";
/// This agent's name, which the store path and the role are built from.
const ENV_AGENT_NAME: &str = "HIVE_AGENT_NAME";
/// The store's address; the rest of its coordinates are `Settings`'s.
const ENV_ADDR: &str = "BAO_ADDR";
/// The store's CA bundle, which may name an undelivered credential.
const ENV_CACERT: &str = "BAO_CACERT";
/// Where this CLI queries, and who it queries as.
pub struct Config {
/// The metrics store's gateway origin, e.g. `https://metrics.example`.
pub base_url: String,
agent: String,
settings: Settings,
role: String,
path: String,
}
impl Config {
/// Read the configuration out of the environment the wrapper sets.
///
/// Errors rather than degrading: an empty result would read as "no such
/// series".
pub fn from_env() -> Result<Self> {
Self::from_lookup(|k| std::env::var(k).ok())
}
fn from_lookup(get: impl Fn(&str) -> Option<String>) -> Result<Self> {
let set = |k: &str| get(k).filter(|v| !v.is_empty());
let Some(base_url) = set(ENV_URL) else {
bail!("this agent has no swarm metrics store configured: {ENV_URL} is unset");
};
if set(ENV_ADDR).is_none() {
bail!(
"{ENV_URL} is set but {ENV_ADDR} is not, so there is no secret store \
to read this agent's credential from — this is a deployment bug"
);
}
let agent =
set(ENV_AGENT_NAME).with_context(|| format!("{ENV_AGENT_NAME} is unset or empty"))?;
let settings =
Settings::from_lookup(|k| set(k).filter(|v| k != ENV_CACERT || ca_is_usable(v)))
.context("reading the swarm secret store's coordinates from the environment")?;
Ok(Self {
base_url,
role: policy::agent_object_name(&agent)?,
path: queue::agent_queue_path(&agent)?,
agent,
settings,
})
}
/// Read this agent's secret from the store and spell it as the bearer.
///
/// Read on every call, so a caller that was refused can call again and
/// present a secret re-minted since.
pub fn bearer(&self) -> Result<String> {
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.context("starting the runtime for the secret store read")?;
let stored: Option<AgentCredential> = runtime.block_on(async {
SecretStore::connect(&self.settings, &self.role, DEFAULT_CERT_MOUNT)
.await
.context("logging in to the swarm secret store with this agent's certificate")?
.read_optional(&self.path)
.await
.with_context(|| format!("reading {}", self.path))
})?;
let stored = stored.with_context(|| format!("nothing is stored at {}", self.path))?;
swarm_queue_client::agent_token::format_agent_token(&self.agent, &stored.value)
.with_context(|| format!("spelling the credential stored at {}", self.path))
}
}
/// Whether the CA bundle at `path` is a file with bytes in it. `BAO_CACERT`
/// names a systemd credential that may not have been delivered; absent means
/// the container's own trust store.
fn ca_is_usable(path: &str) -> bool {
std::fs::metadata(path).is_ok_and(|m| m.len() > 0)
}
#[cfg(test)]
mod tests {
use super::*;
fn env(pairs: &[(&str, &str)]) -> impl Fn(&str) -> Option<String> {
let pairs: Vec<(String, String)> = pairs
.iter()
.map(|(k, v)| ((*k).to_owned(), (*v).to_owned()))
.collect();
move |k| pairs.iter().find(|(n, _)| n == k).map(|(_, v)| v.clone())
}
const FULL: &[(&str, &str)] = &[
(ENV_URL, "https://metrics.example"),
(ENV_AGENT_NAME, "atlas"),
(ENV_ADDR, "https://bao.example:8200"),
("BAO_CLIENT_CERT", "/nonexistent/cert"),
("BAO_CLIENT_KEY", "/nonexistent/key"),
];
#[test]
fn no_url_says_there_is_no_metrics_store() {
let err = Config::from_lookup(env(&[])).err().expect("refused");
assert!(err.to_string().contains("no swarm metrics store"), "{err}");
}
#[test]
fn a_url_without_a_store_is_a_deployment_bug() {
let err = Config::from_lookup(env(&FULL[..2])).err().expect("refused");
assert!(err.to_string().contains("deployment bug"), "{err}");
}
#[test]
fn a_missing_agent_name_is_named() {
let mut pairs = FULL.to_vec();
pairs.retain(|(k, _)| *k != ENV_AGENT_NAME);
let err = Config::from_lookup(env(&pairs)).err().expect("refused");
assert!(err.to_string().contains(ENV_AGENT_NAME), "{err}");
}
/// The path and role are the agent's own, the ones its policy grants.
#[test]
fn the_read_is_the_agents_own_queue_secret() {
let cfg = Config::from_lookup(env(FULL)).expect("configured");
assert_eq!(cfg.base_url, "https://metrics.example");
assert_eq!(cfg.path, "swarm/agents/atlas/queue");
assert_eq!(cfg.role, policy::agent_object_name("atlas").expect("valid"));
}
}

247
swarm-metrics/src/main.rs Normal file
View file

@ -0,0 +1,247 @@
//! `swarm-metrics` — an agent's CLI for the swarm metrics store.
//!
//! The metrics-store sibling of `swarm-logs`: the same credential, the same
//! wrapper shape, and the store's own query language. Each verb is one GET
//! against the store's Prometheus-compatible read API through the
//! gateway, and prints the store's JSON answer unchanged for `jq`.
//!
//! **Read-only.** The verbs map onto `/api/v1/query`, `query_range`,
//! `series`, `labels` and `label/<name>/values`; nothing here writes.
//!
//! ⚠️ **The route applies no scoping.** Any authenticated caller reads the
//! whole swarm's metrics, every hive's.
mod auth;
mod query;
use anyhow::Result;
use clap::{Args, Parser, Subcommand};
use crate::query::{Request, Span};
#[derive(Parser)]
#[command(
name = "swarm-metrics",
version,
about = "query the swarm's metrics store from an agent"
)]
struct Cli {
#[command(subcommand)]
command: Verb,
}
/// Window and cap for the discovery verbs.
#[derive(Args)]
struct SpanArgs {
/// Start of the window: RFC 3339, a unix timestamp, or relative (`-1h`).
///
/// Unset means the store's default: the last day, from 00:00 UTC.
#[arg(long, allow_hyphen_values = true)]
start: Option<String>,
/// End of the window, in the same forms. Unset means now.
#[arg(long, allow_hyphen_values = true)]
end: Option<String>,
/// Return at most this many entries.
#[arg(long)]
limit: Option<u64>,
}
impl From<SpanArgs> for Span {
fn from(a: SpanArgs) -> Self {
Self {
start: a.start,
end: a.end,
limit: a.limit,
}
}
}
#[derive(Subcommand)]
enum Verb {
/// Evaluate a `MetricsQL` (`PromQL`) expression at one instant.
Query {
/// The expression. Quote it: `{`, `|` and `"` are the shell's too.
///
/// OTLP metric names keep their dots, so match them by
/// `{__name__="jobq.nodes"}` rather than as a bare name.
promql: String,
/// The instant: RFC 3339, a unix timestamp, or relative (`-5m`).
/// Unset means now.
#[arg(long, allow_hyphen_values = true)]
time: Option<String>,
},
/// Evaluate an expression over a window, one point per step.
QueryRange {
/// The expression, quoted as for `query`.
promql: String,
/// Start of the window: RFC 3339, a unix timestamp, or relative
/// (`-1h`). Required, so a range is always bounded.
#[arg(long, allow_hyphen_values = true)]
start: String,
/// End of the window, in the same forms. Unset means now.
#[arg(long, allow_hyphen_values = true)]
end: Option<String>,
/// Seconds or a duration (`1m`) between points. Unset means the
/// store's default.
#[arg(long)]
step: Option<String>,
},
/// List the series that match one or more selectors.
Series {
/// A series selector, such as `{"service.name"="nats"}`. Repeatable.
#[arg(required = true)]
matchers: Vec<String>,
#[command(flatten)]
span: SpanArgs,
},
/// List label names, optionally only those on matching series.
Labels {
/// A series selector to narrow to. Repeatable.
#[arg(long = "match")]
matchers: Vec<String>,
#[command(flatten)]
span: SpanArgs,
},
/// List one label's values; `__name__` lists every metric name.
LabelValues {
/// The label name.
name: String,
/// A series selector to narrow to. Repeatable.
#[arg(long = "match")]
matchers: Vec<String>,
#[command(flatten)]
span: SpanArgs,
},
/// Print this CLI's reference as markdown (source of `docs/tools/swarm-metrics-cli.md`).
#[command(hide = true)]
MarkdownDocs,
}
impl Verb {
/// The store request this verb makes; `None` for the local-only verb.
fn request(self) -> Option<Request> {
Some(match self {
Self::Query { promql, time } => Request::Query { promql, time },
Self::QueryRange {
promql,
start,
end,
step,
} => Request::QueryRange {
promql,
start,
end,
step,
},
Self::Series { matchers, span } => Request::Series {
matchers,
span: span.into(),
},
Self::Labels { matchers, span } => Request::Labels {
matchers,
span: span.into(),
},
Self::LabelValues {
name,
matchers,
span,
} => Request::LabelValues {
name,
matchers,
span: span.into(),
},
Self::MarkdownDocs => return None,
})
}
}
fn main() -> Result<()> {
let Some(request) = Cli::parse().command.request() else {
// `show_footer(false)`: same as swarm-logs, the footer is not our prose.
let options = clap_markdown::MarkdownOptions::new().show_footer(false);
print!("{}", clap_markdown::help_markdown_custom::<Cli>(&options));
return Ok(());
};
// Config read only for a store verb: `markdown-docs` runs in a nix check,
// where no credential exists.
swarm_queue_client::install_crypto_provider();
let cfg = auth::Config::from_env()?;
let stdout = std::io::stdout();
query::run(&cfg, &request, &mut stdout.lock())
}
#[cfg(test)]
mod tests {
use super::*;
use clap::CommandFactory as _;
fn parse(args: &[&str]) -> Option<Request> {
Cli::try_parse_from(std::iter::once("swarm-metrics").chain(args.iter().copied()))
.expect("parses")
.command
.request()
}
#[test]
fn the_command_tree_is_well_formed() {
Cli::command().debug_assert();
}
#[test]
fn a_range_query_without_a_start_is_refused() {
assert!(Cli::try_parse_from(["swarm-metrics", "query-range", "up"]).is_err());
}
#[test]
fn series_needs_a_selector() {
assert!(Cli::try_parse_from(["swarm-metrics", "series"]).is_err());
}
#[test]
fn label_values_maps_name_matchers_and_span() {
assert_eq!(
parse(&[
"label-values",
"__name__",
"--match",
"{service.name=\"nats\"}",
"--start",
"-1d",
"--limit",
"5",
]),
Some(Request::LabelValues {
name: "__name__".into(),
matchers: vec!["{service.name=\"nats\"}".into()],
span: Span {
start: Some("-1d".into()),
end: None,
limit: Some(5),
},
})
);
}
#[test]
fn query_range_maps_its_window() {
assert_eq!(
parse(&["query-range", "up", "--start", "-1h", "--step", "1m"]),
Some(Request::QueryRange {
promql: "up".into(),
start: "-1h".into(),
end: None,
step: Some("1m".into()),
})
);
}
#[test]
fn markdown_docs_makes_no_request() {
assert_eq!(parse(&["markdown-docs"]), None);
}
}

355
swarm-metrics/src/query.rs Normal file
View file

@ -0,0 +1,355 @@
//! The one request this binary makes: a read against the metrics store's
//! Prometheus-compatible API on the gateway.
//!
//! The gateway's read locations (`readPaths` in `swarm-victoriametrics.nix`)
//! answer an unauthenticated caller with a bare 401, never a login redirect,
//! so [`run`] treats any non-2xx as a failure with its body rather than as an
//! empty result.
use std::io::Write;
use anyhow::{Context as _, Result, bail};
use reqwest::Url;
use crate::auth::Config;
/// One read against the store, with the store's own parameter names.
#[derive(Debug, PartialEq, Eq)]
pub enum Request {
/// `/api/v1/query`: a `MetricsQL` expression at one instant.
Query {
promql: String,
time: Option<String>,
},
/// `/api/v1/query_range`: a `MetricsQL` expression over a window.
QueryRange {
promql: String,
start: String,
end: Option<String>,
step: Option<String>,
},
/// `/api/v1/series`: the series matching the selectors.
Series { matchers: Vec<String>, span: Span },
/// `/api/v1/labels`: label names.
Labels { matchers: Vec<String>, span: Span },
/// `/api/v1/label/<name>/values`: one label's values.
LabelValues {
name: String,
matchers: Vec<String>,
span: Span,
},
}
/// The window and result cap shared by the discovery endpoints.
#[derive(Debug, Default, PartialEq, Eq)]
pub struct Span {
pub start: Option<String>,
pub end: Option<String>,
pub limit: Option<u64>,
}
/// The full request URL for `request` against the gateway at `base`.
///
/// Pure so every verb's path and parameters are testable without a store.
pub fn url(base: &str, request: &Request) -> Result<Url> {
let mut url = Url::parse(base).with_context(|| format!("{base} is not a metrics store URL"))?;
let mut params: Vec<(&str, &str)> = Vec::new();
let (path, matchers, span): (Vec<&str>, &[String], Option<&Span>) = match request {
Request::Query { promql, time } => {
params.push(("query", promql));
params.extend(time.as_deref().map(|t| ("time", t)));
(vec!["query"], &[], None)
}
Request::QueryRange {
promql,
start,
end,
step,
} => {
params.push(("query", promql));
params.push(("start", start));
params.extend(end.as_deref().map(|t| ("end", t)));
params.extend(step.as_deref().map(|s| ("step", s)));
(vec!["query_range"], &[], None)
}
Request::Series { matchers, span } => (vec!["series"], matchers, Some(span)),
Request::Labels { matchers, span } => (vec!["labels"], matchers, Some(span)),
Request::LabelValues {
name,
matchers,
span,
} => (vec!["label", name, "values"], matchers, Some(span)),
};
params.extend(matchers.iter().map(|m| ("match[]", m.as_str())));
let limit = span.and_then(|s| s.limit).map(|n| n.to_string());
if let Some(span) = span {
params.extend(span.start.as_deref().map(|t| ("start", t)));
params.extend(span.end.as_deref().map(|t| ("end", t)));
params.extend(limit.as_deref().map(|n| ("limit", n)));
}
// `extend` percent-encodes each segment, `/` included.
url.path_segments_mut()
.map_err(|()| anyhow::anyhow!("{base} cannot carry a path"))?
.pop_if_empty()
.extend(["api", "v1"])
.extend(path);
url.query_pairs_mut().extend_pairs(params);
Ok(url)
}
/// Send `request` and copy the store's JSON answer to `out`.
pub fn run(cfg: &Config, request: &Request, out: &mut impl Write) -> Result<()> {
let url = url(&cfg.base_url, request)?;
let http = build_http_client()?;
let send = || -> Result<reqwest::blocking::Response> {
let bearer = cfg
.bearer()
.context("reading this agent's credential for the swarm metrics store")?;
http.get(url.clone())
.bearer_auth(bearer)
.send()
.with_context(|| format!("querying the swarm metrics store at {}", url.path()))
};
let mut response = send()?;
// A 401 can be a secret re-minted between the store read and the check,
// so it is read again and the request sent once more.
if response.status() == reqwest::StatusCode::UNAUTHORIZED {
response = send()?;
}
let status = response.status();
if !status.is_success() {
// nginx answers HTML for an `auth_request` denial, the store answers
// JSON naming the bad expression. Truncated to the first screen.
let body = response.text().unwrap_or_default();
let body = body.trim();
let shown: String = body.chars().take(2000).collect();
bail!(
"the swarm metrics store refused the request ({status}): {shown}{}",
if shown.len() < body.len() {
" […]"
} else {
""
}
);
}
copy_with_newline(&mut response, out)
}
/// Copy `from` to `out` unchanged, ending with exactly one newline when the
/// body did not end in one. Streams: a broad `series` answer can be large.
fn copy_with_newline(from: &mut impl std::io::Read, out: &mut impl Write) -> Result<()> {
let mut buf = [0u8; 8192];
let mut last = b'\n';
loop {
let n = from
.read(&mut buf)
.context("reading the answer from the swarm metrics store")?;
if n == 0 {
break;
}
out.write_all(&buf[..n]).context("writing to stdout")?;
last = buf[n - 1];
}
if last != b'\n' {
writeln!(out).context("writing to stdout")?;
}
Ok(())
}
/// The HTTP client, trusting the container's own roots.
///
/// No timeout: a range query over a long window legitimately takes long, and
/// a client-side cut would read as a failure the store never had.
fn build_http_client() -> Result<reqwest::blocking::Client> {
reqwest::blocking::Client::builder()
.build()
.context("building the metrics-store HTTP client")
}
#[cfg(test)]
mod tests {
use super::*;
const BASE: &str = "https://metrics.example";
fn pairs(u: &Url) -> Vec<(String, String)> {
u.query_pairs()
.map(|(k, v)| (k.into_owned(), v.into_owned()))
.collect()
}
fn owned(p: &[(&str, &str)]) -> Vec<(String, String)> {
p.iter()
.map(|(k, v)| ((*k).to_owned(), (*v).to_owned()))
.collect()
}
#[test]
fn an_instant_query_carries_the_expression_and_time() {
let u = url(
BASE,
&Request::Query {
promql: r#"count by (__name__) ({__name__=~"jobq\\..*"})"#.into(),
time: Some("-5m".into()),
},
)
.expect("builds");
assert_eq!(u.path(), "/api/v1/query");
assert_eq!(
pairs(&u),
owned(&[
("query", r#"count by (__name__) ({__name__=~"jobq\\..*"})"#),
("time", "-5m"),
])
);
}
/// An unset optional parameter is left to the store, not sent empty.
#[test]
fn an_unset_time_is_not_sent() {
let u = url(
BASE,
&Request::Query {
promql: "up".into(),
time: None,
},
)
.expect("builds");
assert_eq!(pairs(&u), owned(&[("query", "up")]));
}
#[test]
fn a_range_query_carries_its_window_and_step() {
let u = url(
BASE,
&Request::QueryRange {
promql: "rate(x[5m])".into(),
start: "-1h".into(),
end: Some("now".into()),
step: Some("1m".into()),
},
)
.expect("builds");
assert_eq!(u.path(), "/api/v1/query_range");
assert_eq!(
pairs(&u),
owned(&[
("query", "rate(x[5m])"),
("start", "-1h"),
("end", "now"),
("step", "1m"),
])
);
}
/// Each selector is its own `match[]`, the Prometheus API's spelling.
#[test]
fn series_sends_every_selector_and_the_span() {
let u = url(
BASE,
&Request::Series {
matchers: vec![r#"{service.name="nats"}"#.into(), "up".into()],
span: Span {
start: Some("-1d".into()),
end: None,
limit: Some(20),
},
},
)
.expect("builds");
assert_eq!(u.path(), "/api/v1/series");
assert_eq!(
pairs(&u),
owned(&[
("match[]", r#"{service.name="nats"}"#),
("match[]", "up"),
("start", "-1d"),
("limit", "20"),
])
);
}
#[test]
fn labels_has_its_own_path() {
let u = url(
BASE,
&Request::Labels {
matchers: vec![],
span: Span::default(),
},
)
.expect("builds");
assert_eq!(u.path(), "/api/v1/labels");
assert_eq!(u.query(), Some(""));
}
/// The label name is a path segment, so one with a dot (an OTLP
/// attribute) stays one segment and a `/` cannot reach another route.
#[test]
fn a_label_name_is_one_encoded_path_segment() {
let u = url(
BASE,
&Request::LabelValues {
name: "service.name".into(),
matchers: vec![],
span: Span::default(),
},
)
.expect("builds");
assert_eq!(u.path(), "/api/v1/label/service.name/values");
let u = url(
BASE,
&Request::LabelValues {
name: "../../admin".into(),
matchers: vec![],
span: Span::default(),
},
)
.expect("builds");
assert_eq!(u.path(), "/api/v1/label/..%2F..%2Fadmin/values");
}
/// A base with a trailing slash or a path prefix still lands on
/// `<base>/api/v1/...`, with no doubled slash.
#[test]
fn the_base_url_keeps_its_prefix_without_a_doubled_slash() {
let q = Request::Query {
promql: "up".into(),
time: None,
};
assert_eq!(
url("https://metrics.example/", &q).expect("builds").path(),
"/api/v1/query"
);
for base in ["https://gw.example/metrics", "https://gw.example/metrics/"] {
assert_eq!(
url(base, &q).expect("builds").path(),
"/metrics/api/v1/query",
"{base}"
);
}
}
#[test]
fn a_relative_base_is_refused() {
let q = Request::Query {
promql: "up".into(),
time: None,
};
assert!(url("metrics.example", &q).is_err());
}
#[test]
fn the_body_is_copied_and_newline_terminated_once() {
let mut out = Vec::new();
copy_with_newline(&mut &br#"{"status":"success"}"#[..], &mut out).expect("copies");
assert_eq!(out, b"{\"status\":\"success\"}\n");
let mut out = Vec::new();
copy_with_newline(&mut &b"{}\n"[..], &mut out).expect("copies");
assert_eq!(out, b"{}\n");
}
}