mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

add swarm-metrics, an agent CLI for the metrics store

Browse source 
swarm-metrics reads the swarm's VictoriaMetrics from inside an agent
container: query, query-range, series, labels and label-values, each one
GET against the store's Prometheus read API through the gateway, printing
the store's JSON. It authenticates the way swarm-logs does, with the
agent's own queue secret read from bao under the agent's leaf, sent as
a swarm-agent bearer.

On the metrics store's host the read API's auth map gains an agent arm
ahead of the Grafana one. It routes to swarm-metrics-agent-auth, a
swarm-logs-auth instance logged in under the existing swarm-metrics-auth
leaf, whose bao grant now includes swarm/agents/+/queue read. The check is
rate-limited per address like the log store's
(deploy.victoriametrics.agentAuth{Rate,Burst,VerdictTtl}).

nix/agent-modules/metrics.nix installs the wrapped binary when the hive
forwards HIVE_AGENT_METRICS_URL (from swarm.victoriametrics.domain) and a
store address, the same gate as swarm-logs. Adds the base:swarm-metrics
skill and the CLI reference with its freshness check.

Fixes #5395
This commit is contained in:
atlas 2026-10-17 00:55:46 +02:00 • committed by mara
commit e92f118bda
30 changed files with 1417 additions and 88 deletions
Download patch file Download diff file Expand all files Collapse all files

1
.prettierignore
View file

@ -8,6 +8,7 @@ docs/tools/hivectl-cli.md
docs/tools/swarmctl-cli.md docs/tools/swarmctl-cli.md
docs/tools/forge-cli.md docs/tools/forge-cli.md
docs/tools/swarm-logs-cli.md docs/tools/swarm-logs-cli.md
docs/tools/swarm-metrics-cli.md
# Files with multi-line list-item continuations that prettier strips to col 0. # Files with multi-line list-item continuations that prettier strips to col 0.
# prettier's `proseWrap: "preserve"` prevents prose reflow but not list-item # prettier's `proseWrap: "preserve"` prevents prose reflow but not list-item

3
CLAUDE.md
View file

@ -138,10 +138,11 @@ Swarm-level:
token against the store; shared by `swarm-nats-auth` and token against the store; shared by `swarm-nats-auth` and
`swarm-logs-auth`. `swarm-logs-auth`.
- **`swarm-logs-auth/`** — nginx `auth_request` responder for agent access - **`swarm-logs-auth/`** — nginx `auth_request` responder for agent access
to the log store. to the log store, and to the metrics store as `swarm-metrics-agent-auth`.
- **`swarm-metrics-auth/`** — nginx `auth_request` responder for Grafana's - **`swarm-metrics-auth/`** — nginx `auth_request` responder for Grafana's
bearer on the metrics store's query API. bearer on the metrics store's query API.
- **`swarm-logs/`** — agent CLI: a LogsQL query in, log lines out. - **`swarm-logs/`** — agent CLI: a LogsQL query in, log lines out.
- **`swarm-metrics/`** — agent CLI: a MetricsQL query in, the store's JSON out.
- **`swarm-matrix-client/`** — appservice calls against the swarm - **`swarm-matrix-client/`** — appservice calls against the swarm
homeserver, shared by `swarm-controller` and `swarm-matrix-ctl`. homeserver, shared by `swarm-controller` and `swarm-matrix-ctl`.
- **`swarm-matrix-ctl/`** — the one binary run inside - **`swarm-matrix-ctl/`** — the one binary run inside

13
Cargo.lock generated
View file

@ -4944,6 +4944,19 @@ dependencies = [
"tracing-subscriber", "tracing-subscriber",
] ]
[[package]]
name = "swarm-metrics"
version = "0.1.0"
dependencies = [
"anyhow",
"clap",
"clap-markdown",
"reqwest",
"swarm-queue-client",
"swarm-secret-client",
"tokio",
]
[[package]] [[package]]
name = "swarm-metrics-auth" name = "swarm-metrics-auth"
version = "0.1.0" version = "0.1.0"

1
Cargo.toml
View file

@ -38,6 +38,7 @@ members = [
"swarm-queue-client", "swarm-queue-client",
"swarm-review-classifier", "swarm-review-classifier",
"swarm-logs", "swarm-logs",
"swarm-metrics",
"swarm-logs-auth", "swarm-logs-auth",
"swarm-metrics-auth", "swarm-metrics-auth",
"swarm-secret-client", "swarm-secret-client",

50
claude-plugins/plugins/base/skills/swarm-metrics/SKILL.md Normal file
View file

@ -0,0 +1,50 @@
---
name: swarm-metrics
description: Query the swarm metrics store with `swarm-metrics` when you need to know whether a service's metrics reach VictoriaMetrics, which metric names or labels exist, or what a counter or gauge did over a window. Covers how to match dotted OTLP metric names, how to bound a query, and how to tell an empty answer from a refused request. Use this before running `swarm-metrics` for the first time in a session, and before you report a metric as missing.
---
# swarm-metrics
`swarm-metrics` reads the swarm's VictoriaMetrics through the gateway as you,
the agent. Every verb prints the store's JSON answer, so pipe it into `jq`.
## Find out what exists first
Metric names from OTLP keep their dots (`jobq.nodes`,
`hyperhive.vcs.commit.count`), and so do OTLP attribute labels
(`service.name`). Match a dotted metric by `__name__`, and quote a dotted
label name:
```console
$ swarm-metrics label-values __name__ --match '{__name__=~"jobq\\..*"}' | jq -r '.data[]'
$ swarm-metrics label-values service.name | jq -r '.data[]'
$ swarm-metrics series '{"service.name"="nats"}' --start -1h --limit 20
```
`series`, `labels` and `label-values` cover the last day from 00:00 UTC
unless you pass `--start`/`--end`. Use `--limit` on a broad selector.
## Query
```console
$ swarm-metrics query 'count by (__name__) ({__name__=~"hyperhive\\.vcs\\..*"})'
$ swarm-metrics query-range 'sum(rate({__name__="jobq.nodes"}[5m]))' --start -6h --step 5m
```
Times take RFC 3339, a unix timestamp or a relative duration (`-1h`).
`query-range` requires `--start`, so a range is always bounded. Inside the
expression, a regex needs its dots escaped twice (`\\.`), once for the
PromQL string and once for the regex.
## Telling "no series" from "refused"
- **No matching series**: exit 0, and `.data.result` (or `.data`) is `[]`.
- **Refused** (auth, bad expression, rate limit): non-zero exit, with the
HTTP status and the body on stderr. A 403 is the gateway's rate limit.
## Before you report a metric missing, run a control
An empty answer fits a wrong name, a missing label value, or a window too
narrow just as well as a metric that never arrived. Query something that must
exist, such as `label-values service.name`, and widen `--start`. If the
control is also empty, the query is wrong, not the store.

2
docs/process/gotchas.md
View file

@ -83,7 +83,7 @@ With both set, systemd creates the runtime directory under `/run/private/`
nginx's `connect()` to a socket in it fails with `EACCES` whatever the nginx's `connect()` to a socket in it fails with `EACCES` whatever the
directory and socket modes are. A unit serving a socket to another user directory and socket modes are. A unit serving a socket to another user
runs as a static user instead (`swarm-controller`, `swarm-logs-auth`, runs as a static user instead (`swarm-controller`, `swarm-logs-auth`,
`swarm-metrics-auth`). `swarm-metrics-auth`, `swarm-metrics-agent-auth`).
`module-eval-core-toggle` fails on any `unix:` upstream of nginx owned by `module-eval-core-toggle` fails on any `unix:` upstream of nginx owned by
such a unit. such a unit.

4
docs/swarm/credentials.md
View file

@ -56,7 +56,7 @@ of the cell says how.
<!-- vale write-good.Passive = NO --> <!-- vale write-good.Passive = NO -->
| store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull | | store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull |
| ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `swarm/agents/<agent>/matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again | | `swarm/agents/<agent>/matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again |
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | ✅ a secondary that didn't come up at start, or whose token the homeserver rejected, is read from the store again two minutes later, then at doubling waits up to sixteen minutes, without restarting the daemon or its other accounts; one that's up and then rejected ends its sync loop, the daemon exits and its timer restarts it | | `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | ✅ a secondary that didn't come up at start, or whose token the homeserver rejected, is read from the store again two minutes later, then at doubling waits up to sixteen minutes, without restarting the daemon or its other accounts; one that's up and then rejected ends its sync loop, the daemon exits and its timer restarts it |
| `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase for every account listed under the agent's `matrix/`, in a five-minute pass | the agent container itself, under the certificate its hive passed in, when `hive-matrix-daemon` brings the account up | ❌ never replaced: the account's server-side key backup opens only with the passphrase it was enabled with, so the pass writes only to an empty path | ❌ read when the account comes up; an account whose passphrase lands later turns backup on at the next daemon start | | `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase for every account listed under the agent's `matrix/`, in a five-minute pass | the agent container itself, under the certificate its hive passed in, when `hive-matrix-daemon` brings the account up | ❌ never replaced: the account's server-side key backup opens only with the passphrase it was enabled with, so the pass writes only to an empty path | ❌ read when the account comes up; an account whose passphrase lands later turns backup on at the next daemon start |
@ -65,7 +65,7 @@ of the cell says how.
| `swarm/controller/swarm-controller/forge/token` | the forge's `forgejo-swarm-controller-account`, in the `hive-forge` container, once, under the forge publisher's certificate with a write-only grant; it then revokes its earlier tokens | `swarm-controller`, under its own certificate, at start and on refusal | ❌ the forge mints it once and keeps only the token's name in the container, at `/var/lib/forgejo/swarm-controller-token-name`; deleting that file mints a new token on the unit's next start | ✅ the controller reads it at start, and in the background with backoff until it first finds it. When the forge answers 401 it reads it again and sends the request once more with the new token, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value | | `swarm/controller/swarm-controller/forge/token` | the forge's `forgejo-swarm-controller-account`, in the `hive-forge` container, once, under the forge publisher's certificate with a write-only grant; it then revokes its earlier tokens | `swarm-controller`, under its own certificate, at start and on refusal | ❌ the forge mints it once and keeps only the token's name in the container, at `/var/lib/forgejo/swarm-controller-token-name`; deleting that file mints a new token on the unit's next start | ✅ the controller reads it at start, and in the background with backoff until it first finds it. When the forge answers 401 it reads it again and sends the request once more with the new token, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value |
| `swarm/controller/swarm-controller/grafana/token` | `swarm-bao-grafana-controller-token` on Grafana's host, under Grafana's certificate: a token for the `swarm-controller` Viewer service account, minted with Grafana's admin password | `swarm-controller`, under its own certificate, on the first `/api/alerts` request and again when Grafana refuses it | ✅ a timer starts the unit again 10 minutes after each run; it keeps a stored token Grafana accepts, and mints a new one when Grafana answers 401 or the store holds none | ✅ the controller holds the token in memory; when Grafana answers 401 it reads the store once more and retries the request | | `swarm/controller/swarm-controller/grafana/token` | `swarm-bao-grafana-controller-token` on Grafana's host, under Grafana's certificate: a token for the `swarm-controller` Viewer service account, minted with Grafana's admin password | `swarm-controller`, under its own certificate, on the first `/api/alerts` request and again when Grafana refuses it | ✅ a timer starts the unit again 10 minutes after each run; it keeps a stored token Grafana accepts, and mints a new one when Grafana answers 401 or the store holds none | ✅ the controller holds the token in memory; when Grafana answers 401 it reads the store once more and retries the request |
| `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config; then `hive-agent-bao-identity` in the agent container, under the leaf it holds | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ✅ `hive-agent-bao-identity` re-reads it every 10 minutes and swaps a new leaf into `/run/hive-agent-bao-identity/`, which every store login in the container presents. It reads under the leaf it holds, so it renews only while that leaf is valid | | `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config; then `hive-agent-bao-identity` in the agent container, under the leaf it holds | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ✅ `hive-agent-bao-identity` re-reads it every 10 minutes and swaps a new leaf into `/run/hive-agent-bao-identity/`, which every store login in the container presents. It reads under the leaf it holds, so it renews only while that leaf is valid |
| `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive; `swarm-logs` in the agent container, under the same certificate, on every query, as the bearer it presents at the log store; `swarm-nats-auth` and `swarm-logs-auth`, each under its own certificate, to check a presented secret | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff. `swarm-logs` reads it on every query and once more after a 401. `swarm-logs-auth` keeps granting a secret it verified for `deploy.victorialogs.authVerdictTtl` (60 s by default) without reading the path, so a replaced or deleted secret still works at the log store for up to that long | | `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive; `swarm-logs` and `swarm-metrics` in the agent container, under the same certificate, on every query, as the bearer they present at the log and metrics stores; `swarm-nats-auth` and `swarm-logs-auth`, each under its own certificate, and `swarm-metrics-agent-auth` under `swarm-metrics-auth`'s, to check a presented secret | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff. `swarm-logs` and `swarm-metrics` read it on every query and once more after a 401. `swarm-logs-auth` keeps granting a secret it verified for `deploy.victorialogs.authVerdictTtl` (60 s by default) without reading the path, so a replaced or deleted secret still works at the log store for up to that long; `swarm-metrics-agent-auth` likewise at the metrics store, for `deploy.victoriametrics.agentAuthVerdictTtl` |
| `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer | | `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer |
| `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when an operator links an external forge account in the swarm UI | `hive-agent-forge-accounts` in the agent container, under the agent's own certificate, into `<state>/forge-<label>-token` and `forge-<label>.json` | ❌ an operator's token; replaced only by deleting it and linking the label again | ✅ the agent re-fetches on a 2-minute timer | | `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when an operator links an external forge account in the swarm UI | `hive-agent-forge-accounts` in the agent container, under the agent's own certificate, into `<state>/forge-<label>-token` and `forge-<label>.json` | ❌ an operator's token; replaced only by deleting it and linking the label again | ✅ the agent re-fetches on a 2-minute timer |
| `swarm/agents/<agent>/github-token` | `swarm-controller`, when an operator links a GitHub account in the swarm UI | `hive-agent-github-token` in the agent container, under the agent's own certificate, into `<state>/github-token` | ❌ an operator's token; replaced only by deleting it and linking it again | ✅ the agent re-fetches on a 2-minute timer, and `hive-github-notify` re-reads the file on every poll | | `swarm/agents/<agent>/github-token` | `swarm-controller`, when an operator links a GitHub account in the swarm UI | `hive-agent-github-token` in the agent container, under the agent's own certificate, into `<state>/github-token` | ❌ an operator's token; replaced only by deleting it and linking it again | ✅ the agent re-fetches on a 2-minute timer, and `hive-github-notify` re-reads the file on every poll |

28
docs/swarm/services.md
View file

@ -1,3 +1,7 @@
| `deploy.victoriametrics.agentAuthRate` | Default `50r/s` per client address, for agent reads checked by `swarm-metrics-agent-auth`; over it and `agentAuthBurst`, the gateway answers 403. |
| `deploy.victoriametrics.agentAuthBurst` | Default `500` requests above `agentAuthRate` before the 403s start. |
| `deploy.victoriametrics.agentAuthVerdictTtl` | Default `60` seconds that `swarm-metrics-agent-auth` keeps granting an agent secret it verified. A replaced or deleted secret keeps working at the metrics store for up to that long. |
# Swarm-wide services # Swarm-wide services
Some things exist once per **swarm** rather than once per hive: the forge, Some things exist once per **swarm** rather than once per hive: the forge,
@ -201,11 +205,14 @@ VictoriaMetrics authenticates nothing itself; the `metrics.<swarm-domain>`
vhost does it: vhost does it:
- **The read API** (the Prometheus query, series, label and metadata - **The read API** (the Prometheus query, series, label and metadata
routes): Grafana's datasource or an operator. Grafana sends a bearer that routes): Grafana's datasource, an agent, or an operator. Grafana sends a
`swarm-bao-grafana-metrics-bearer` creates in the swarm secret store, and bearer that `swarm-bao-grafana-metrics-bearer` creates in the swarm secret
`swarm-metrics-auth` on the store's host checks it against the same store store, and `swarm-metrics-auth` on the store's host checks it against the
path. A request without a bearer needs an operator's authelia session same store path. An agent's `swarm-metrics` sends its
(group `admins`); without one it gets a 401, not a login redirect. `swarm-agent.<agent>.<secret>` bearer, which `swarm-metrics-agent-auth` on
the store's host checks against `swarm/agents/<agent>/queue`. A request
without a bearer needs an operator's authelia session (group `admins`);
without one it gets a 401, not a login redirect.
- **vmui**: an operator's authelia session, with a redirect to the login. - **vmui**: an operator's authelia session, with a redirect to the login.
The bare domain redirects to vmui. The bare domain redirects to vmui.
- **The collector's OTLP push, and its scrape of the store's own - **The collector's OTLP push, and its scrape of the store's own
@ -213,13 +220,16 @@ vhost does it:
any other session or client there, operators included. any other session or client there, operators included.
- **Every other path**, the store's write and admin API included: 403. - **Every other path**, the store's write and admin API included: 403.
`swarm-metrics-auth` reads the store with a leaf of its own, `swarm-metrics-auth` and `swarm-metrics-agent-auth` read the store with one
`deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`. leaf, `deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`.
Where the secret store runs on the same host, these default to the leaf it Where the secret store runs on the same host, these default to the leaf it
mints. On any other host you place a leaf whose CN is mints. On any other host you place a leaf whose CN is
`deploy.bao.metricsAuthCommonName`, and evaluation refuses the config until `deploy.bao.metricsAuthCommonName`, and evaluation refuses the config until
you set both. It reads the stored bearer at most once a minute, so a you set both. `swarm-metrics-auth` reads the stored bearer at most once a
replaced bearer starts working up to a minute later. minute, so a replaced bearer starts working up to a minute later.
`swarm-metrics-agent-auth` is the log store's `swarm-logs-auth` binary, with
the same grant TTL and rate limit, under `deploy.victoriametrics.agentAuth*`
(table above).
The store's container has no network, so its journal forwarder leaves through The store's container has no network, so its journal forwarder leaves through
a relay. Inside the container, `swarm-victoriametrics-egress-relay` listens on a relay. Inside the container, `swarm-victoriametrics-egress-relay` listens on

123
docs/tools/swarm-metrics-cli.md Normal file
View file

@ -0,0 +1,123 @@
# Command-Line Help for `swarm-metrics`
This document contains the help content for the `swarm-metrics` command-line program.
**Command Overview:**
* [`swarm-metrics`↴](#swarm-metrics)
* [`swarm-metrics query`↴](#swarm-metrics-query)
* [`swarm-metrics query-range`↴](#swarm-metrics-query-range)
* [`swarm-metrics series`↴](#swarm-metrics-series)
* [`swarm-metrics labels`↴](#swarm-metrics-labels)
* [`swarm-metrics label-values`↴](#swarm-metrics-label-values)
## `swarm-metrics`
query the swarm's metrics store from an agent
**Usage:** `swarm-metrics <COMMAND>`
###### **Subcommands:**
* `query` — Evaluate a `MetricsQL` (`PromQL`) expression at one instant
* `query-range` — Evaluate an expression over a window, one point per step
* `series` — List the series that match one or more selectors
* `labels` — List label names, optionally only those on matching series
* `label-values` — List one label's values; `__name__` lists every metric name
## `swarm-metrics query`
Evaluate a `MetricsQL` (`PromQL`) expression at one instant
**Usage:** `swarm-metrics query [OPTIONS] <PROMQL>`
###### **Arguments:**
* `<PROMQL>` — The expression. Quote it: `{`, `|` and `"` are the shell's too.
OTLP metric names keep their dots, so match them by `{__name__="jobq.nodes"}` rather than as a bare name.
###### **Options:**
* `--time <TIME>` — The instant: RFC 3339, a unix timestamp, or relative (`-5m`). Unset means now
## `swarm-metrics query-range`
Evaluate an expression over a window, one point per step
**Usage:** `swarm-metrics query-range [OPTIONS] --start <START> <PROMQL>`
###### **Arguments:**
* `<PROMQL>` — The expression, quoted as for `query`
###### **Options:**
* `--start <START>` — Start of the window: RFC 3339, a unix timestamp, or relative (`-1h`). Required, so a range is always bounded
* `--end <END>` — End of the window, in the same forms. Unset means now
* `--step <STEP>` — Seconds or a duration (`1m`) between points. Unset means the store's default
## `swarm-metrics series`
List the series that match one or more selectors
**Usage:** `swarm-metrics series [OPTIONS] <MATCHERS>...`
###### **Arguments:**
* `<MATCHERS>` — A series selector, such as `{"service.name"="nats"}`. Repeatable
###### **Options:**
* `--start <START>` — Start of the window: RFC 3339, a unix timestamp, or relative (`-1h`).
Unset means the store's default: the last day, from 00:00 UTC.
* `--end <END>` — End of the window, in the same forms. Unset means now
* `--limit <LIMIT>` — Return at most this many entries
## `swarm-metrics labels`
List label names, optionally only those on matching series
**Usage:** `swarm-metrics labels [OPTIONS]`
###### **Options:**
* `--match <MATCHERS>` — A series selector to narrow to. Repeatable
* `--start <START>` — Start of the window: RFC 3339, a unix timestamp, or relative (`-1h`).
Unset means the store's default: the last day, from 00:00 UTC.
* `--end <END>` — End of the window, in the same forms. Unset means now
* `--limit <LIMIT>` — Return at most this many entries
## `swarm-metrics label-values`
List one label's values; `__name__` lists every metric name
**Usage:** `swarm-metrics label-values [OPTIONS] <NAME>`
###### **Arguments:**
* `<NAME>` — The label name
###### **Options:**
* `--match <MATCHERS>` — A series selector to narrow to. Repeatable
* `--start <START>` — Start of the window: RFC 3339, a unix timestamp, or relative (`-1h`).
Unset means the store's default: the last day, from 00:00 UTC.
* `--end <END>` — End of the window, in the same forms. Unset means now
* `--limit <LIMIT>` — Return at most this many entries

11
docs/trust-boundary/security.md
View file

@ -84,12 +84,13 @@ A hive reads no **swarm service's** OIDC client secret. Every reader of
collector and the store forwarder each fetch theirs with a leaf of their own, collector and the store forwarder each fetch theirs with a leaf of their own,
and the store's granter fetches the bao UI's. and the store's granter fetches the bao UI's.
Two services read **every agent's queue secret** Three services read **every agent's queue secret**
(`swarm/agents/<agent>/queue`), each under a leaf of its own whose policy (`swarm/agents/<agent>/queue`), each under a leaf of its own whose policy
reaches that one leaf per agent and nothing else: the queue's auth callout reaches that one leaf per agent: the queue's auth callout (`swarm-nats-auth`),
(`swarm-nats-auth`) and the log store's agent check (`swarm-logs-auth`, on the the log store's agent check (`swarm-logs-auth`, on the log store's host), and
log store's host). Either one compromised can present any agent's queue the metrics store's (`swarm-metrics-agent-auth`, under the `swarm-metrics-auth`
secret, at the queue and at the log store's query route alike. leaf on the metrics store's host). Any one compromised can present any agent's
queue secret, at the queue and at both stores' read routes alike.
### Per-hive credentials live under the hive kind ### Per-hive credentials live under the hive kind

1
flake.nix
View file

@ -96,6 +96,7 @@
hive-screen-mcp hive-screen-mcp
hive-subagent-daemon hive-subagent-daemon
swarm-logs swarm-logs
swarm-metrics
assets assets
frontend frontend
reference-docs reference-docs

26
hive-c0re/src/meta.rs
View file

@ -387,6 +387,8 @@ const FORWARDED_VARS: &[&str] = &[
// same reason as the two above: it is a gateway address, and a container // same reason as the two above: it is a gateway address, and a container
// cannot derive it from anything it holds. // cannot derive it from anything it holds.
"HIVE_AGENT_LOGS_QUERY_URL", "HIVE_AGENT_LOGS_QUERY_URL",
// The metrics store's origin, for `swarm-metrics`, for the same reason.
"HIVE_AGENT_METRICS_URL",
"HIVE_FORGE_URL", "HIVE_FORGE_URL",
"HIVE_FORGE_PUBLIC_URL", "HIVE_FORGE_PUBLIC_URL",
"HIVE_MATRIX_URL", "HIVE_MATRIX_URL",
@ -433,6 +435,11 @@ const FORWARDED_VAR_OPTIONS: &[(&str, &str)] = &[
"HIVE_AGENT_LOGS_QUERY_URL", "HIVE_AGENT_LOGS_QUERY_URL",
"services.hyperhive.agent.logs.queryUrl", "services.hyperhive.agent.logs.queryUrl",
), ),
// The same for `swarm-metrics`: see `nix/agent-modules/metrics.nix`.
(
"HIVE_AGENT_METRICS_URL",
"services.hyperhive.agent.metrics.url",
),
("HIVE_FORGE_URL", "services.hyperhive.agent.forge.url"), ("HIVE_FORGE_URL", "services.hyperhive.agent.forge.url"),
("HIVE_MATRIX_URL", "services.hyperhive.agent.matrix.url"), ("HIVE_MATRIX_URL", "services.hyperhive.agent.matrix.url"),
("HYPERHIVE_SWARM_NAME", "services.hyperhive.agent.swarmName"), ("HYPERHIVE_SWARM_NAME", "services.hyperhive.agent.swarmName"),
@ -1644,6 +1651,25 @@ mod tests {
); );
} }
/// `swarm-metrics` is installed only when the option is set, and reads
/// the env var at runtime, so the metrics store's origin needs both halves.
#[test]
fn the_metrics_url_is_forwarded_as_both_env_and_option() {
assert!(FORWARDED_VARS.contains(&"HIVE_AGENT_METRICS_URL"));
let mut out = String::new();
push_forwarded_var_options(
&mut out,
&[(
"HIVE_AGENT_METRICS_URL",
"https://metrics.t.local".to_string(),
)],
);
assert_eq!(
out,
" services.hyperhive.agent.metrics.url = \"https://metrics.t.local\";\n"
);
}
#[test] #[test]
fn require_service_urls_accepts_a_rendered_forge_url() { fn require_service_urls_accepts_a_rendered_forge_url() {
require_service_urls(&[ require_service_urls(&[

1
nix/agent-modules/default.nix
View file

@ -48,6 +48,7 @@ in
./matrix.nix ./matrix.nix
./mcp.nix ./mcp.nix
./memory-cap.nix ./memory-cap.nix
./metrics.nix
./network.nix ./network.nix
./otel.nix ./otel.nix
./otel-removed.nix ./otel-removed.nix

66
nix/agent-modules/metrics.nix Normal file
View file

@ -0,0 +1,66 @@
# `swarm-metrics` on this agent's PATH, configured for this swarm's metrics
# store. The sibling of ./logs.nix, in the same shape for the same reasons:
# the binary defaults nothing (see `swarm-metrics/README.md`), so it is
# installed wrapped, and it is gated on the secret store's address as well as
# on the store URL, because the identity it presents is the agent's own queue
# secret read from that store.
{
lib,
pkgs,
config,
...
}:
let
cfg = config.services.hyperhive.agent.metrics;
baoCfg = config.services.hyperhive.agent.bao;
configured = cfg.url != null && baoCfg.addr != null;
# The coordinates ./logs.nix gives `swarm-logs`. Paths and a name only: a
# value in `--set` lands in the world-readable nix store.
wrapperEnv = {
HIVE_AGENT_METRICS_URL = cfg.url;
HIVE_AGENT_NAME = config.services.hyperhive.agent.user.name;
BAO_ADDR = baoCfg.addr;
BAO_CLIENT_CERT = baoCfg.clientCertFile;
BAO_CLIENT_KEY = baoCfg.clientKeyFile;
BAO_CACERT = "/run/credentials/hive-agent.service/hive-agent-bao-server-ca";
};
swarmMetricsConfigured = pkgs.symlinkJoin {
name = "swarm-metrics-configured";
paths = [ config.services.hyperhive.agent.packages.swarm-metrics ];
nativeBuildInputs = [ pkgs.makeWrapper ];
postBuild = ''
wrapProgram $out/bin/swarm-metrics ${
lib.concatStringsSep " " (
lib.mapAttrsToList (name: value: "--set ${name} ${lib.escapeShellArg value}") wrapperEnv
)
}
'';
};
in
{
options.services.hyperhive.agent.metrics = {
url = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "https://metrics.example.com";
description = ''
The swarm metrics store's gateway origin, as this container reaches
it. `swarm-metrics` appends the read API's paths (`/api/v1/query`,
...) to it.
Set by the generated meta flake from the host's own
{option}`services.hyperhive.swarm.victoriametrics.domain`.
`null` means this hive has not been given the metrics store's address
for its agents, and `swarm-metrics` is then not installed at all.
Not defaulted: a guessed address resolves cleanly to the wrong place.
'';
};
};
config = lib.mkIf configured {
environment.systemPackages = [ swarmMetricsConfigured ];
};
}

15
nix/checks.nix
View file

@ -478,6 +478,21 @@ in
touch "$out" touch "$out"
''; '';
# `swarm-metrics` CLI reference freshness check, the same shape as
# `swarm-logs-docs` above.
swarm-metrics-docs =
pkgs.runCommand "swarm-metrics-docs-fresh" { nativeBuildInputs = [ pkgs.diffutils ]; }
''
${self.packages.${system}.swarm-metrics}/bin/swarm-metrics markdown-docs > generated.md
if ! diff -u ${../docs/tools/swarm-metrics-cli.md} generated.md; then
echo "" >&2
echo "ERROR: docs/tools/swarm-metrics-cli.md is out of date — regenerate it:" >&2
echo " nix build .#swarm-metrics && ./result/bin/swarm-metrics markdown-docs > docs/tools/swarm-metrics-cli.md" >&2
exit 1
fi
touch "$out"
'';
# `hive-forge` CLI reference freshness check — same shape as # `hive-forge` CLI reference freshness check — same shape as
# `hivectl-docs` above, `hive-forge markdown-docs` (clap-markdown over # `hivectl-docs` above, `hive-forge markdown-docs` (clap-markdown over
# its own command tree) instead. Reuses `packages.<system>.hive-forge` # its own command tree) instead. Reuses `packages.<system>.hive-forge`

4
nix/host-modules/glue-bao-tls.nix
View file

@ -285,8 +285,8 @@ in
[ -s ${pkiDir}/logs-auth.pem ] || ${signLeaf} ${pkiDir} logs-auth \ [ -s ${pkiDir}/logs-auth.pem ] || ${signLeaf} ${pkiDir} logs-auth \
${lib.escapeShellArg deployCfg.bao.logsAuthCommonName} "" clientAuth ${lib.escapeShellArg deployCfg.bao.logsAuthCommonName} "" clientAuth
# The metrics store's Grafana auth responder, which reads Grafana's # The metrics store's auth responders, which read Grafana's metrics
# metrics bearer. Minted here for the matrix-ctl leaf's reason: the # bearer and agents' queue secrets. Minted here for the matrix-ctl leaf's reason: the
# metrics store is a swarm singleton, so elsewhere this is the file an # metrics store is a swarm singleton, so elsewhere this is the file an
# operator copies. # operator copies.
[ -s ${pkiDir}/metrics-auth.pem ] || ${signLeaf} ${pkiDir} metrics-auth \ [ -s ${pkiDir}/metrics-auth.pem ] || ${signLeaf} ${pkiDir} metrics-auth \

4
nix/host-modules/glue-metrics-auth-bao-identity.nix
View file

@ -1,5 +1,5 @@
# Glue: point the metrics store's Grafana auth responder at the bao leaf minted # Glue: point the metrics store's auth responders (Grafana's and the agents')
# for it. # at the bao leaf minted for them.
# #
# ONE PAIRING PER FILE — the swarm-metrics-auth principal ← bao, and nothing # ONE PAIRING PER FILE — the swarm-metrics-auth principal ← bao, and nothing
# else. Deleting this leaves a metrics store with no responder identity unless # else. Deleting this leaves a metrics store with no responder identity unless

3
nix/host-modules/hive-c0re/environment.nix
View file

@ -287,6 +287,9 @@ in
# `swarm-victorialogs.nix`'s location for the whole reasoning. # `swarm-victorialogs.nix`'s location for the whole reasoning.
lib.optionalAttrs haveBaoClientIdentity { lib.optionalAttrs haveBaoClientIdentity {
HIVE_AGENT_LOGS_QUERY_URL = "https://${config.services.hyperhive.swarm.victorialogs.domain}/select/logsql/query"; HIVE_AGENT_LOGS_QUERY_URL = "https://${config.services.hyperhive.swarm.victorialogs.domain}/select/logsql/query";
# The metrics store's origin, for `swarm-metrics`; same credential and
# gate. Its read locations answer a bare 401, never a login redirect.
HIVE_AGENT_METRICS_URL = "https://${config.services.hyperhive.swarm.victoriametrics.domain}";
} }
// //
# Where the swarm's secret store is, and the identity this hive presents to # Where the swarm's secret store is, and the identity this hive presents to

16
nix/host-modules/swarm-bao.nix
View file

@ -1060,15 +1060,18 @@ let
} }
]; ];
# The metrics store's responder, which checks the bearer Grafana presents # The metrics store's responders under one leaf: the agent one checks the
# against the one stored for it; and the host unit that fetches the store # secret an agent presents against the one stored for it (the same `+` leaf
# container's journal forwarder secret under the same leaf. Nothing else. # as the log store's), the other checks the bearer Grafana presents; and the
# host unit that fetches the store container's journal forwarder secret.
# Nothing else.
metricsAuthReaders = [ metricsAuthReaders = [
{ {
name = "swarm-metrics-auth"; name = "swarm-metrics-auth";
cn = baoDeploy.metricsAuthCommonName; cn = baoDeploy.metricsAuthCommonName;
policyText = policyText =
readStanza "${credentialMountPath}/data/${grafanaMetricsBearerLeaf}" readStanza "${credentialMountPath}/data/swarm/agents/+/queue"
+ readStanza "${credentialMountPath}/data/${grafanaMetricsBearerLeaf}"
# The metrics store container's journal forwarder, fetched on its host. # The metrics store container's journal forwarder, fetched on its host.
+ readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.victoriametrics.otel.clientId}/oidc/client"; + readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.victoriametrics.otel.clientId}/oidc/client";
} }
@ -1977,8 +1980,9 @@ in
default = "swarm-metrics-auth"; default = "swarm-metrics-auth";
description = '' description = ''
Subject the store's `swarm-metrics-auth` cert-auth role accepts: the Subject the store's `swarm-metrics-auth` cert-auth role accepts: the
identity the metrics store's Grafana auth responder presents to read identity the metrics store's auth responders present to read agent
the bearer Grafana's datasource sends, queue credentials, `swarm/agents/<agent>/queue`, and the bearer
Grafana's datasource sends,
`swarm/services/<grafana client id>/metrics/bearer`, and the one `swarm/services/<grafana client id>/metrics/bearer`, and the one
`swarm-victoriametrics-forwarder-oidc` presents to read `swarm-victoriametrics-forwarder-oidc` presents to read
`swarm/services/<metrics store forwarder client id>/oidc/client`. `swarm/services/<metrics store forwarder client id>/oidc/client`.

135
nix/host-modules/swarm-victoriametrics.nix
View file

@ -38,9 +38,9 @@ let
nginxGid = config.ids.gids.nginx; nginxGid = config.ids.gids.nginx;
# The store's read API, which the vhost forwards to a caller holding # The store's read API, which the vhost forwards to a caller holding
# Grafana's bearer or an operator's session. Each location's `proxy_pass` # Grafana's bearer, an agent's own bearer, or an operator's session. Each
# carries the path, so the store receives the path nginx matched, not the # location's `proxy_pass` carries the path, so the store receives the path
# caller's unnormalised spelling. # nginx matched, not the caller's unnormalised spelling.
# #
# Grafana's Prometheus datasource reads the query, series, label, metadata, # Grafana's Prometheus datasource reads the query, series, label, metadata,
# buildinfo, exemplar, rules and alerts routes; the store answers the last # buildinfo, exemplar, rules and alerts routes; the store answers the last
@ -128,6 +128,14 @@ let
# same leaf as `grafanaMetricsBearerLeaf` in ./swarm-bao.nix. # same leaf as `grafanaMetricsBearerLeaf` in ./swarm-bao.nix.
grafanaBearerStorePath = "swarm/services/${hyperhiveCfg.swarm.grafana.oidc.clientId}/metrics/bearer"; grafanaBearerStorePath = "swarm/services/${hyperhiveCfg.swarm.grafana.oidc.clientId}/metrics/bearer";
# The agents' check on the read API: a `swarm-logs-auth` binary, logging in
# under this host's `swarm-metrics-auth` leaf, whose grant covers every
# agent's queue secret. The log store's host runs `swarm-metrics-auth` under
# its `swarm-logs-auth` leaf the same way.
agentAuthUnit = "swarm-metrics-agent-auth";
agentAuthSocket = "/run/${agentAuthUnit}/auth.sock";
agentAuthCredential = id: "/run/credentials/${agentAuthUnit}.service/${id}";
# ── the container's journal forwarder ─────────────────────────────── # ── the container's journal forwarder ───────────────────────────────
# #
# With no interface but loopback, the forwarder reaches nothing by name. A # With no interface but loopback, the forwarder reaches nothing by name. A
@ -242,7 +250,8 @@ in
example = "/var/lib/swarm-bao-pki/metrics-auth.pem"; example = "/var/lib/swarm-bao-pki/metrics-auth.pem";
description = '' description = ''
Client certificate `swarm-metrics-auth` presents to the swarm's secret Client certificate `swarm-metrics-auth` presents to the swarm's secret
store to read Grafana's metrics bearer. Its subject must be store to read Grafana's metrics bearer, and `swarm-metrics-agent-auth`
to read agents' queue secrets. Its subject must be
{option}`services.hyperhive.deploy.bao.metricsAuthCommonName`. {option}`services.hyperhive.deploy.bao.metricsAuthCommonName`.
Required when `deploy.victoriametrics.enable` is set; unset fails at Required when `deploy.victoriametrics.enable` is set; unset fails at
@ -264,6 +273,47 @@ in
''; '';
}; };
options.services.hyperhive.deploy.victoriametrics.agentAuthVerdictTtl = lib.mkOption {
type = lib.types.ints.unsigned;
default = 60;
description = ''
Seconds `swarm-metrics-agent-auth` grants an agent bearer it has
verified again without reading the secret store. A denial is never
remembered.
This is also how long a re-minted or deleted queue secret keeps working
at the metrics store's read API after its last successful check. `0`
checks the store on every query.
'';
};
options.services.hyperhive.deploy.victoriametrics.agentAuthRate = lib.mkOption {
type = lib.types.str;
default = "50r/s";
example = "30r/m";
description = ''
nginx `limit_req` rate, per client address, of read API requests
carrying a `swarm-agent.` bearer, which nginx checks with
`swarm-metrics-agent-auth`. Each such request that the responder has
not granted within
{option}`services.hyperhive.deploy.victoriametrics.agentAuthVerdictTtl`
costs a secret store login and read, whether the bearer is right or not.
Keyed by client address, so every agent on one host shares one budget.
Requests over the rate and its burst are answered 403.
'';
};
options.services.hyperhive.deploy.victoriametrics.agentAuthBurst = lib.mkOption {
type = lib.types.ints.positive;
default = 500;
description = ''
Requests a client address may make above
{option}`services.hyperhive.deploy.victoriametrics.agentAuthRate` before
nginx answers 403, served without delay.
'';
};
config = lib.mkIf deployCfg.victoriametrics.enable { config = lib.mkIf deployCfg.victoriametrics.enable {
assertions = [ assertions = [
{ {
@ -287,7 +337,8 @@ in
services.hyperhive.deploy.victoriametrics.authBaoClientCertFile and services.hyperhive.deploy.victoriametrics.authBaoClientCertFile and
authBaoClientKeyFile must both be set. authBaoClientKeyFile must both be set.
swarm-metrics-auth reads Grafana's metrics bearer from the swarm's swarm-metrics-auth reads Grafana's metrics bearer, and
swarm-metrics-agent-auth agents' queue secrets, from the swarm's
secret store with this certificate. They default to the leaf the secret store with this certificate. They default to the leaf the
store mints where services.hyperhive.deploy.bao.enable is set on store mints where services.hyperhive.deploy.bao.enable is set on
this host; elsewhere, issue a leaf whose CN is this host; elsewhere, issue a leaf whose CN is
@ -320,14 +371,18 @@ in
# Which check a read location's `auth_request` gets. `auth_request` takes a # Which check a read location's `auth_request` gets. `auth_request` takes a
# fixed URI, so it names `/__metrics_read_auth`, which rewrites to the # fixed URI, so it names `/__metrics_read_auth`, which rewrites to the
# target this map picks: a bearer goes to Grafana's check, anything else # target this map picks: an agent's `swarm-agent.` bearer goes to the
# (an operator's session cookie) to authelia. After the forwarder's egress # agents' check, any other bearer to Grafana's, anything else (an
# operator's session cookie) to authelia. A map tries its regexes in
# order, so the agent arm comes first. After the forwarder's egress
# listener, `egressServer`. # listener, `egressServer`.
services.nginx.appendHttpConfig = egressServer + '' services.nginx.appendHttpConfig = egressServer + ''
map $http_authorization $swarm_metrics_read_auth_target { map $http_authorization $swarm_metrics_read_auth_target {
"~^Bearer swarm-agent\." /__metrics_agent_auth;
"~^Bearer " /__metrics_grafana_auth; "~^Bearer " /__metrics_grafana_auth;
default /__metrics_authelia; default /__metrics_authelia;
} }
limit_req_zone $binary_remote_addr zone=swarm_metrics_agent_auth:1m rate=${deployCfg.victoriametrics.agentAuthRate};
''; '';
# `forceSSL`: authelia answers a plain-http auth subrequest # `forceSSL`: authelia answers a plain-http auth subrequest
@ -400,6 +455,22 @@ in
''; '';
}; };
# An agent's bearer, checked by `swarm-metrics-agent-auth`. Any
# well-formed agent bearer the responder has not recently granted
# costs a store login and read, so the rate per client address is
# capped here. 403 when over: `auth_request` passes only 401 and 403
# through, and `swarm-metrics` retries a 401.
"= /__metrics_agent_auth" = {
proxyPass = "http://unix:${agentAuthSocket}:/";
extraConfig = ''
internal;
limit_req zone=swarm_metrics_agent_auth burst=${toString deployCfg.victoriametrics.agentAuthBurst} nodelay;
limit_req_status 403;
proxy_pass_request_body off;
proxy_set_header Content-Length "";
'';
};
# The authelia subrequest. For the collector's two locations above it # The authelia subrequest. For the collector's two locations above it
# admits only the collector's client; for the read and vmui locations # admits only the collector's client; for the read and vmui locations
# it admits an operator's session (`metricsStoreRules` in # it admits an operator's session (`metricsStoreRules` in
@ -506,6 +577,56 @@ in
}; };
}; };
# The agents' check on the read API, see `agentAuthUnit`.
users.users.${agentAuthUnit} = {
isSystemUser = true;
group = agentAuthUnit;
description = "swarm metrics store auth_request responder for agent credentials";
};
users.groups.${agentAuthUnit} = { };
systemd.services.${agentAuthUnit} = {
description = "swarm metrics store auth_request responder for agent credentials";
wantedBy = [ "multi-user.target" ];
after = [
"network.target"
"swarm-bao-pki.service"
];
serviceConfig = {
ExecStart = lib.concatStringsSep " " [
"${deployCfg.victorialogs.authPackage}/bin/swarm-logs-auth"
"--socket ${agentAuthSocket}"
"--store-cert-role swarm-metrics-auth"
"--verdict-ttl ${toString deployCfg.victoriametrics.agentAuthVerdictTtl}"
];
LoadCredential = lib.mapAttrsToList (id: path: "${id}:${path}") metricsAuthStoreFiles;
# ⚠️ A static user, never `DynamicUser`: see the `swarm-metrics-auth` unit above.
User = agentAuthUnit;
Group = agentAuthUnit;
RuntimeDirectory = agentAuthUnit;
RuntimeDirectoryMode = "0751";
RuntimeDirectoryPreserve = "yes";
Restart = "on-failure";
RestartSec = "5s";
SyslogIdentifier = agentAuthUnit;
ProtectSystem = "strict";
ProtectHome = true;
PrivateTmp = "disconnected";
PrivateDevices = true;
NoNewPrivileges = true;
RestrictSUIDSGID = true;
RemoveIPC = true;
};
environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
BAO_CLIENT_CERT = agentAuthCredential "bao-client.pem";
BAO_CLIENT_KEY = agentAuthCredential "bao-client-key.pem";
}
// lib.optionalAttrs (metricsAuthStoreFiles ? "bao-ca.pem") {
BAO_CACERT = agentAuthCredential "bao-ca.pem";
};
};
# nginx runs as its own user under `ProtectSystem=strict`, so it can only # nginx runs as its own user under `ProtectSystem=strict`, so it can only
# create the socket where it may write. It removes the socket on a clean # create the socket where it may write. It removes the socket on a clean
# exit only; a stale one after a crash fails every later `bind`, and with # exit only; a stale one after a crash fails every later `bind`, and with

Some files were not shown because too many files have changed in this diff Show more