From 7d09776742939ee54656c1c6a9d5df20253bf7ec Mon Sep 17 00:00:00 2001 From: atlas Date: Thu, 15 Oct 2026 12:26:06 +0200 Subject: [PATCH] swarm UI: an alerts page, read by the controller with a Viewer Grafana token from bao swarm-bao-grafana-controller-token, on the host that runs Grafana, logs in to Grafana's API over its socket as the built-in admin (the password swarm-bao-grafana-admin minted, #5138) and keeps a `swarm-controller` service account that is an enabled Viewer. It mints a token for that account and writes it to bao at swarm/controller/swarm-controller/grafana/token under Grafana's leaf. A ten-minute timer runs it again. The unit keeps a stored token Grafana accepts. If Grafana refuses the stored token with 401, or bao holds none, the unit deletes every token of the account and mints a new one. Grafana's grant gains create/read/update on that leaf, and the controller's gains read. swarm-controller serves GET /api/alerts from Grafana's /api/prometheus/grafana/api/v1/alerts. It reads the token from bao, keeps it in memory, and re-reads bao once on a 401. The HTTP client has 5s connect and 15s request timeouts. Any failure is a 503 that names its cause. swarm-ui gets an /alerts page listing them, built the same way as the hives page. This is option (c) on #5129, the operator's choice. --- docs/swarm/secrets.md | 1 + docs/swarm/ui.md | 23 +- frontend/packages/swarm-ui/src/App.tsx | 2 + .../swarm-ui/src/pages/AlertsPage.tsx | 138 +++++++ .../packages/swarm-ui/src/shell/Shell.tsx | 1 + .../glue-bao-readers-policy-order.nix | 7 +- .../glue-bao-readers-store-order.nix | 1 + nix/host-modules/swarm-bao.nix | 26 +- nix/host-modules/swarm-controller.nix | 5 + nix/host-modules/swarm-grafana.nix | 150 +++++++ nix/module-eval/bao-controller.nix | 9 +- nix/module-eval/bao-grants.nix | 65 ++- nix/module-eval/grafana.nix | 27 ++ swarm-controller/README.md | 3 + swarm-controller/src/agent_limits.rs | 1 + swarm-controller/src/alerts.rs | 391 ++++++++++++++++++ swarm-controller/src/forge_account.rs | 1 + swarm-controller/src/github_account.rs | 1 + swarm-controller/src/linked_accounts.rs | 1 + swarm-controller/src/main.rs | 8 + swarm-controller/src/matrix_account.rs | 1 + swarm-secret-client/src/grafana.rs | 54 +++ swarm-secret-client/src/lib.rs | 9 +- 23 files changed, 888 insertions(+), 37 deletions(-) create mode 100644 frontend/packages/swarm-ui/src/pages/AlertsPage.tsx create mode 100644 swarm-controller/src/alerts.rs create mode 100644 swarm-secret-client/src/grafana.rs diff --git a/docs/swarm/secrets.md b/docs/swarm/secrets.md index fbc1faee..00a5799d 100644 --- a/docs/swarm/secrets.md +++ b/docs/swarm/secrets.md @@ -60,6 +60,7 @@ neither is a renaming of the other. | a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | | the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf** | `/var/lib/hive-otel-oidc/hive-.secret` on the hive's host, 0400 root, loaded by `LoadCredential` | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | | the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | `/var/lib/forgejo/swarm-controller-token` inside the forge container, the minter's record that it has minted; `swarm-bao-forge-publisher` copies it into the swarm secret store at `swarm/controller/swarm-controller/forge/token` each time the forge container starts and each time the mint writes the file; the controller holds it in memory only | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) | +| the swarm controller's Grafana token | `swarm-bao-grafana-controller-token.service`, **on the host that runs Grafana**, as Grafana's built-in admin over its socket: it creates the `swarm-controller` service account as a Viewer (and sets it back to Viewer on every run), mints a token for it and writes it to the swarm secret store at `swarm/controller/swarm-controller/grafana/token` under Grafana's leaf. Re-run every ten minutes: it keeps a stored token Grafana accepts, and replaces one Grafana refuses with 401 (or a missing one), deleting every other token of that account first | the swarm secret store only; the controller reads it for `GET /api/alerts` and holds it in memory, re-reading the store once when Grafana answers 401 | nothing to place: Grafana's leaf is already required to run Grafana. Until the unit has run, `/api/alerts` answers 503 naming the empty path | | an agent's claude login (`.credentials.json`) | `swarm-claude-login.service`, in its container, when an operator completes `claude auth login` from the agent's card in the swarm UI; it stores the file as a new version at `swarm/agents//claude-credentials` under its own leaf, which may create and update that path and read nothing under an agent | the swarm secret store at that path; the agent's `hive-agent` reads it under the agent's own leaf every 30 seconds while it waits for a login and writes it to `~/.claude/.credentials.json` | the same read, co-located or not; nothing for an operator to place beyond the agent's store leaf. When the store's copy and the agent's differ the KV version decides: the agent installs only a version newer than the last it installed (`claude-credentials-version` under the harness dir), because claude rotates the refresh token in its own copy and the stored one goes stale. The agent ignores a stale or equal version, and a relogin writes a newer one — [claude-login.md](claude-login.md) | | the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | | authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia | diff --git a/docs/swarm/ui.md b/docs/swarm/ui.md index 606819a9..4bdec1a9 100644 --- a/docs/swarm/ui.md +++ b/docs/swarm/ui.md @@ -7,17 +7,18 @@ domain, covers host-level detail for one hive. ## What it shows -| route | what | -| ---------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | -| `/` | the start page: the swarm's name over a digital-rain backdrop; the swarm name in the header links here | -| `/hives` | the hive directory, each hive with its last reported status | -| `/agents` | every agent: status, config PR, wanted state, inference settings, resource caps; create agents, link forge, matrix and GitHub accounts | -| `/agents//terminal` | one agent's live terminal | -| `/agents//subagents//terminal` | one of that agent's subagents' live terminal | -| `/jobs` | the controller's job graph — where agent creation and credential mints show progress | -| `/schedules` | the scheduled prompts: add, edit, pause, resume and remove | -| `/issues` | a cross-repo issue report | -| `/review` | the review classifier: its recent runs, a try-out on a real PR, and its settings | +| route | what | +| ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | +| `/` | the start page: the swarm's name over a digital-rain backdrop; the swarm name in the header links here | +| `/hives` | the hive directory, each hive with its last reported status | +| `/agents` | every agent: status, config PR, wanted state, inference settings, resource caps; create agents, link forge, matrix and GitHub accounts | +| `/agents//terminal` | one agent's live terminal | +| `/agents//subagents//terminal` | one of that agent's subagents' live terminal | +| `/alerts` | the alerts the swarm's Grafana holds now (firing, pending, no data, error), read through the controller with a Viewer service-account token | +| `/jobs` | the controller's job graph — where agent creation and credential mints show progress | +| `/schedules` | the scheduled prompts: add, edit, pause, resume and remove | +| `/issues` | a cross-repo issue report | +| `/review` | the review classifier: its recent runs, a try-out on a real PR, and its settings | Everything it shows comes from [`swarm-controller`](../../swarm-controller/README.md). An agent created here or with `swarmctl agent create` starts `paused`; set it diff --git a/frontend/packages/swarm-ui/src/App.tsx b/frontend/packages/swarm-ui/src/App.tsx index 5fac588c..619028c6 100644 --- a/frontend/packages/swarm-ui/src/App.tsx +++ b/frontend/packages/swarm-ui/src/App.tsx @@ -4,6 +4,7 @@ import { Route, Switch } from "wouter-preact"; import { Shell } from "./shell/Shell.js"; import { AgentsPage } from "./pages/agents/AgentsPage.js"; import { AgentTerminalPage } from "./pages/agents/AgentTerminalPage.js"; +import { AlertsPage } from "./pages/AlertsPage.js"; import { ComponentsPage } from "./pages/ComponentsPage.js"; import { JobsPage } from "./pages/JobsPage.js"; import { HivesPage } from "./pages/HivesPage.js"; @@ -27,6 +28,7 @@ export function App() { + — `/alerts`' content: the alerts the swarm's Grafana holds +// now, from swarm-controller's `GET /api/alerts`. Same shape as `HivesPage`: +// one fetch on a `RefreshIntervalPicker` cadence, rendered as a `Table`. +import { useState } from "preact/hooks"; +import { ApiErrorPanel } from "@hive/shared/api-error-panel.js"; +import { readApiError, type ProblemDetails } from "@hive/shared/api-error.js"; +import { Badge, type BadgeTone } from "@hive/shared/badge.js"; +import { Panel } from "../ui/panel/Panel.js"; +import { RelativeTime } from "../ui/relative-time/RelativeTime.js"; +import { + RefreshIntervalPicker, + useRefreshInterval, + type RefreshIntervalMs, +} from "../ui/refresh-interval/RefreshInterval.js"; +import { Table, type TableColumn } from "../ui/table/Table.js"; + +interface Alert { + name: string; + // Grafana's state for the instance: `Alerting`, `Pending`, `NoData` or + // `Error`. Anything else renders neutral. + state: string; + active_at: string | null; + summary: string | null; + labels: Record; +} + +const STATE_TONE: Record = { + Alerting: "negative", + Error: "negative", + NoData: "warning", + Pending: "warning", +}; + +// Grafana reports an instance that never became active with the zero time, +// which `Date.parse` reads as year 1. +function activeSinceMs(a: Alert): number | null { + if (a.active_at === null) return null; + const ms = Date.parse(a.active_at); + return Number.isNaN(ms) || ms <= 0 ? null : ms; +} + +// A row needs a key that is unique among instances of the same rule. +function alertKey(a: Alert): string { + return JSON.stringify(Object.entries(a.labels).sort()); +} + +const COLUMNS: TableColumn[] = [ + { + key: "name", + header: "alert", + render: (a) => a.name, + sortBy: (a) => a.name, + filterValue: (a) => a.name, + }, + { + key: "state", + header: "state", + render: (a) => { + const since = activeSinceMs(a); + return ( + + {a.state} () + + ) : ( + a.state + ) + } + /> + ); + }, + sortBy: (a) => a.state, + filterValues: (a) => [a.state], + filterMode: "multiselect", + }, + { + key: "summary", + header: "summary", + render: (a) => a.summary ?? "—", + sortBy: (a) => a.summary ?? "", + filterValue: (a) => a.summary ?? "", + }, +]; + +const DEFAULT_INTERVAL_MS: RefreshIntervalMs = 30_000; + +export function AlertsPage() { + const [alerts, setAlerts] = useState(null); + const [error, setError] = useState(null); + const [intervalMs, setIntervalMs] = + useState(DEFAULT_INTERVAL_MS); + + useRefreshInterval(intervalMs, () => { + (async () => { + const r = await fetch("/api/alerts"); + if (!r.ok) { + setError(await readApiError(r)); + return; + } + setAlerts((await r.json()) as Alert[]); + setError(null); + })().catch((e: unknown) => setError({ detail: String(e) })); + }); + + return ( + + } + > + {error ? ( + + ) : null} + {!error && alerts === null ?

loading…

: null} + {alerts && alerts.length === 0 ?

no alerts.

: null} + {alerts && alerts.length > 0 ? ( + + ) : null} + + ); +} diff --git a/frontend/packages/swarm-ui/src/shell/Shell.tsx b/frontend/packages/swarm-ui/src/shell/Shell.tsx index 05cbcc46..d7ffab96 100644 --- a/frontend/packages/swarm-ui/src/shell/Shell.tsx +++ b/frontend/packages/swarm-ui/src/shell/Shell.tsx @@ -51,6 +51,7 @@ const MOTION_KEY = "swarm-ui:motion-override"; const NAV_ITEMS: { href: string; label: string; accent: string }[] = [ { href: "/hives", label: "hives", accent: "var(--purple)" }, { href: "/agents", label: "agents", accent: "var(--green)" }, + { href: "/alerts", label: "alerts", accent: "var(--red)" }, { href: "/jobs", label: "jobs", accent: "var(--pink)" }, { href: "/schedules", label: "schedules", accent: "var(--cyan)" }, { href: "/issues", label: "issues", accent: "var(--yellow)" }, diff --git a/nix/host-modules/glue-bao-readers-policy-order.nix b/nix/host-modules/glue-bao-readers-policy-order.nix index 90412d41..76e3824b 100644 --- a/nix/host-modules/glue-bao-readers-policy-order.nix +++ b/nix/host-modules/glue-bao-readers-policy-order.nix @@ -67,7 +67,8 @@ let ]; }; # ./swarm-grafana.nix, under Grafana's leaf, which may create the admin - # password and the metrics bearer only once that policy is written. + # password, the metrics bearer and the controller's Grafana token only + # once that policy is written. swarm-bao-grafana-admin = { here = deployCfg.grafana.enable; policies = [ "swarm-bao-grafana-oidc-policy" ]; @@ -76,6 +77,10 @@ let here = deployCfg.grafana.enable; policies = [ "swarm-bao-grafana-oidc-policy" ]; }; + swarm-bao-grafana-controller-token = { + here = deployCfg.grafana.enable; + policies = [ "swarm-bao-grafana-oidc-policy" ]; + }; # ./hive-matrix.nix, under matrix-ctl's leaf. swarm-bao-matrix-oidc = { here = deployCfg.matrix.enable; diff --git a/nix/host-modules/glue-bao-readers-store-order.nix b/nix/host-modules/glue-bao-readers-store-order.nix index 0e74d121..b320d7dc 100644 --- a/nix/host-modules/glue-bao-readers-store-order.nix +++ b/nix/host-modules/glue-bao-readers-store-order.nix @@ -47,6 +47,7 @@ let swarm-bao-grafana-oidc = deployCfg.grafana.enable; swarm-bao-grafana-admin = deployCfg.grafana.enable; swarm-bao-grafana-metrics-bearer = deployCfg.grafana.enable; + swarm-bao-grafana-controller-token = deployCfg.grafana.enable; # ./swarm-nats.nix swarm-bao-nats-callout-xkey = deployCfg.nats.enable; # ./swarm-ci-runner.nix diff --git a/nix/host-modules/swarm-bao.nix b/nix/host-modules/swarm-bao.nix index 1d77b11c..77b788ef 100644 --- a/nix/host-modules/swarm-bao.nix +++ b/nix/host-modules/swarm-bao.nix @@ -369,9 +369,9 @@ let # Each hive's matrix sender token (`matrix_account::hive_sender`) grants `read` # for the same keep-if-live reason, and `+` for the same glob-only-as-last-segment reason. # - # Three leaves it reads and never writes: the swarm appservice token - # (matrix-ctl, `matrixCtlPolicyText` below), its own OIDC client secret (the - # publisher) and its own forge token (the forge's host, `forgePublisherWriters`). + # Four leaves it reads and never writes: the swarm appservice token (matrix-ctl), + # its OIDC client secret (the publisher), its forge token (the forge's host) + # and its Grafana token (Grafana's host). Each writer's grant is named below. # # The agent PKI grant is its only path on that mount: it can ask the one role # for a certificate, not write that role or reach the issuer. @@ -457,6 +457,10 @@ let capabilities = ["read"] } + path "${credentialMountPath}/data/${controllerGrafanaTokenLeaf}" { + capabilities = ["read"] + } + path "${agentPkiMountPath}/issue/${agentPkiRoleName}" { capabilities = ["update"] } @@ -582,6 +586,11 @@ let # it; the controller reads it and holds no other copy. controllerForgeTokenLeaf = "swarm/controller/swarm-controller/forge/token"; + # The controller's Viewer token for Grafana's API, the nix half of + # `swarm_secret_client::grafana::controller_token_path`. Grafana's host + # writes it; the controller reads it and holds no other copy. + controllerGrafanaTokenLeaf = "swarm/controller/swarm-controller/grafana/token"; + # The forge's runner registration token, the nix half of # `swarm_secret_client::ci::registration_token_path`. The controller writes # it and cannot read it back (`forge::runner_token` keeps the value it last @@ -911,6 +920,10 @@ let # in ./swarm-grafana.nix) and the bearer its metrics datasource presents # (`swarm-bao-grafana-metrics-bearer`). `create` without `update`, so a stored # value is never replaced. + # + # The controller's Grafana token is the one it may replace: + # `swarm-bao-grafana-controller-token` mints a new one when Grafana refuses + # the stored one, as it does after its database is recreated. grafanaOidcReaders = [ { name = "swarm-grafana-oidc"; @@ -918,7 +931,12 @@ let policyText = readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.grafana.oidc.clientId}/oidc/client" + createStanza "${credentialMountPath}/data/${grafanaAdminLeaf}" - + createStanza "${credentialMountPath}/data/${grafanaMetricsBearerLeaf}"; + + createStanza "${credentialMountPath}/data/${grafanaMetricsBearerLeaf}" + + '' + path "${credentialMountPath}/data/${controllerGrafanaTokenLeaf}" { + capabilities = ["create", "read", "update"] + } + ''; } ]; diff --git a/nix/host-modules/swarm-controller.nix b/nix/host-modules/swarm-controller.nix index da205d18..68c5b864 100644 --- a/nix/host-modules/swarm-controller.nix +++ b/nix/host-modules/swarm-controller.nix @@ -811,6 +811,11 @@ in # separated: a matrix user id cannot contain a space. Consumed by # swarm-controller/src/matrix_account/operator_admin.rs. SWARM_CONTROLLER_MATRIX_ADMIN_SYNC_IGNORE_USERS = lib.concatStringsSep " " matrixCfg.adminSync.ignoreUsers; + # Where `GET /api/alerts` reads alerts (swarm-controller/src/alerts.rs). + # Every swarm has one Grafana, reached by name wherever it runs. The + # token is in the store, written by `swarm-bao-grafana-controller-token` + # on Grafana's host. + SWARM_CONTROLLER_GRAFANA_URL = "https://${swarmCfg.grafana.domain}"; } // queueEnv // forgeEnv diff --git a/nix/host-modules/swarm-grafana.nix b/nix/host-modules/swarm-grafana.nix index 84d10961..d3a16c05 100644 --- a/nix/host-modules/swarm-grafana.nix +++ b/nix/host-modules/swarm-grafana.nix @@ -273,6 +273,14 @@ let metricsBearerPath = "/var/lib/grafana-metrics/bearer"; hostMetricsBearerPath = "/var/lib/nixos-containers/${cfg.machine}${metricsBearerPath}"; + # swarm-controller's token for Grafana's API, minted by + # `swarm-bao-grafana-controller-token` below for a Viewer service account of + # this name. `controllerGrafanaTokenLeaf` in ./swarm-bao.nix is the nix half + # of the path, `swarm_secret_client::grafana::controller_token_path` the Rust + # half. + controllerAccountName = "swarm-controller"; + controllerTokenStorePath = "secret/swarm/controller/swarm-controller/grafana/token"; + socketPath = "${deployCfg.grafana.socketDir}/grafana.sock"; # Both static NixOS ids, and both checked rather than assumed: nginx has @@ -849,6 +857,148 @@ in ''; }; + # swarm-controller reads Grafana's alerts with a token of a Viewer service + # account, created here through Grafana's API as the built-in admin and + # handed over through the store. + # + # Re-run by the timer below. A stored token Grafana still accepts is kept; + # one it refuses with 401, or none at all, is replaced: every token of the + # account is deleted and one new one is minted and stored. The account's + # role is set back to Viewer on every run. + systemd.services.swarm-bao-grafana-controller-token = { + description = "give swarm-controller a Viewer token for Grafana's API through the swarm secret store"; + # Ordered after the store where it runs here: + # ./glue-bao-readers-store-order.nix. + after = [ + "container@${cfg.machine}.service" + "swarm-bao-grafana-admin.service" + ]; + wantedBy = [ "multi-user.target" ]; + path = [ + baoDeploy.package + pkgs.coreutils + pkgs.curl + pkgs.jq + ]; + # ./lib/store-retry.nix. A Grafana still starting refuses the + # connection, and that attempt fails into the same retry. + inherit (storeRetry) startLimitBurst startLimitIntervalSec; + serviceConfig = storeRetry.serviceConfig // { + # No `RemainAfterExit`: the timer below starts this again. + Type = "oneshot"; + SyslogIdentifier = "swarm-bao-grafana-controller-token"; + # Grafana's answers, the new token among them, pass through a temp file. + PrivateTmp = true; + # Each call below is bounded by curl's own timeouts; a run makes up to + # about ten of them. + TimeoutStartSec = "3min"; + }; + environment = { + BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; + BAO_CLIENT_CERT = baoDeploy.grafanaOidcClientCertFile; + BAO_CLIENT_KEY = baoDeploy.grafanaOidcClientKeyFile; + } + // lib.optionalAttrs (baoDeploy.serverCaFile != null) { + BAO_CACERT = baoDeploy.serverCaFile; + }; + script = '' + set -euo pipefail + + err="$(mktemp)" + body="$(mktemp)" + trap 'rm -f "$err" "$body"' EXIT + + # One call to Grafana's API through its socket. Prints the HTTP + # status; the body lands in "$body". The credential reaches curl as + # a header file, never as an argument. + grafana() { + local auth="$1" method="$2" path="$3" + shift 3 + curl --silent --show-error --connect-timeout 5 --max-time 15 \ + --unix-socket ${lib.escapeShellArg socketPath} \ + -H @<(printf 'Authorization: %s\n' "$auth") \ + -H 'Content-Type: application/json' \ + -X "$method" -o "$body" -w '%{http_code}' "$@" \ + "http://localhost$path" + } + + # Fails the attempt unless the last call answered one of the statuses + # given. + expect() { + local what="$1" status="$2" + shift 2 + for ok in "$@"; do + [ "$status" = "$ok" ] && return 0 + done + echo "Grafana answered $status to $what:" >&2 + cat "$body" >&2 + exit 1 + } + + if [ ! -s ${lib.escapeShellArg hostAdminPasswordPath} ]; then + echo "no Grafana admin password at ${hostAdminPasswordPath} yet; swarm-bao-grafana-admin writes it." >&2 + exit 1 + fi + admin="Basic $(printf 'admin:%s' "$(< ${lib.escapeShellArg hostAdminPasswordPath})" | base64 -w0)" + + status="$(grafana "$admin" GET '/api/serviceaccounts/search?perpage=1000&query=${controllerAccountName}')" + expect "the service account search" "$status" 200 + account="$(jq -c --arg n ${controllerAccountName} '[.serviceAccounts[] | select(.name == $n)][0] // empty' "$body")" + if [ -z "$account" ]; then + status="$(grafana "$admin" POST /api/serviceaccounts \ + --data '{"name":"${controllerAccountName}","role":"Viewer","isDisabled":false}')" + expect "creating the ${controllerAccountName} service account" "$status" 200 201 + account="$(jq -c . "$body")" + echo "created the ${controllerAccountName} service account as a Viewer" + fi + id="$(jq -r .id <<<"$account")" + if [ "$(jq -r '"\(.role) \(.isDisabled // false)"' <<<"$account")" != "Viewer false" ]; then + status="$(grafana "$admin" PATCH "/api/serviceaccounts/$id" \ + --data '{"role":"Viewer","isDisabled":false}')" + expect "setting the ${controllerAccountName} service account to an enabled Viewer" "$status" 200 + echo "set the ${controllerAccountName} service account back to an enabled Viewer" + fi + + if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then + echo "could not log in to swarm-bao with Grafana's certificate; leaving swarm-controller's Grafana token as it is." >&2 + cat "$err" >&2 + exit 1 + fi + export BAO_TOKEN + + # Checked against the call the controller makes with it. + if stored="$(bao kv get -field=value ${lib.escapeShellArg controllerTokenStorePath} 2>"$err")" && [ -n "$stored" ]; then + status="$(grafana "Bearer $stored" GET /api/prometheus/grafana/api/v1/alerts)" + case "$status" in + 200) exit 0 ;; + 401) echo "Grafana refuses the stored token for ${controllerAccountName}; replacing it." ;; + *) expect "reading alerts with the stored token" "$status" 200 ;; + esac + fi + + status="$(grafana "$admin" GET "/api/serviceaccounts/$id/tokens")" + expect "listing the ${controllerAccountName} tokens" "$status" 200 + for token_id in $(jq -r '.[].id' "$body"); do + status="$(grafana "$admin" DELETE "/api/serviceaccounts/$id/tokens/$token_id")" + expect "deleting token $token_id of ${controllerAccountName}" "$status" 200 + done + + status="$(grafana "$admin" POST "/api/serviceaccounts/$id/tokens" \ + --data "{\"name\":\"${controllerAccountName}-$(date +%s)\"}")" + expect "minting a ${controllerAccountName} token" "$status" 200 + key="$(jq -r .key "$body")" + : >"$body" + if ! printf '%s' "$key" | bao kv put ${lib.escapeShellArg controllerTokenStorePath} value=- >/dev/null 2>"$err"; then + echo "could not store the new ${controllerAccountName} token at ${controllerTokenStorePath}; the next run replaces it." >&2 + cat "$err" >&2 + exit 1 + fi + echo "stored a new ${controllerAccountName} token at ${controllerTokenStorePath}" + ''; + }; + + systemd.timers.swarm-bao-grafana-controller-token = storeRefreshTimer "check swarm-controller's Grafana token and replace it if Grafana refuses it"; + containers.${cfg.machine} = { autoStart = true; ephemeral = false; diff --git a/nix/module-eval/bao-controller.nix b/nix/module-eval/bao-controller.nix index d5c83d9c..982d6749 100644 --- a/nix/module-eval/bao-controller.nix +++ b/nix/module-eval/bao-controller.nix @@ -221,10 +221,10 @@ let && builtins.elem "bao-client-key.pem:/var/lib/swarm-bao-pki/controller-key.pem" s.swarm-controller.serviceConfig.LoadCredential; } { - # The forge token comes from the store, so the unit is handed no file for - # it, and neither fixture runs the forge: a controller reaches it by name - # wherever it runs. - name = "the controller is handed no forge token file, and is pointed at the forge away from it"; + # The forge and Grafana tokens come from the store, so the unit is handed + # no file for either, and neither fixture runs the forge or Grafana: a + # controller reaches both by name wherever they run. + name = "the controller is handed no forge token file, and is pointed at the forge and Grafana away from it"; ok = lib.all ( @@ -235,6 +235,7 @@ let !(lib.any (c: lib.hasPrefix "forge-token:" c) u.serviceConfig.LoadCredential) && !(u.environment ? SWARM_CONTROLLER_FORGE_TOKEN_FILE) && u.environment.SWARM_CONTROLLER_FORGE_URL == "https://forge.t.local" + && u.environment.SWARM_CONTROLLER_GRAFANA_URL == "https://grafana.t.local" ) [ baoControllerHere diff --git a/nix/module-eval/bao-grants.nix b/nix/module-eval/bao-grants.nix index 989d2542..8481a4ac 100644 --- a/nix/module-eval/bao-grants.nix +++ b/nix/module-eval/bao-grants.nix @@ -597,8 +597,10 @@ let # # The admin password and the metrics bearer are the leaves it may create, # and without `update`: a grant that could replace one would let a second - # run rotate a value Grafana or the metrics store already holds. - name = "the Grafana reader reads its own client secret and creates its admin password and metrics bearer, and nothing else"; + # run rotate a value Grafana or the metrics store already holds. The + # controller's Grafana token is the one leaf it may replace, which it + # does when Grafana refuses the stored token. + name = "the Grafana reader reads its own client secret, creates its admin password and metrics bearer, and writes the controller's Grafana token, and nothing else"; ok = let s = baoGrantHere.systemd.services.swarm-bao-grafana-oidc-policy.script; @@ -622,6 +624,14 @@ let "read" ]; } + { + path = "secret/data/swarm/controller/swarm-controller/grafana/token"; + caps = [ + "create" + "read" + "update" + ]; + } ] && !(lib.hasInfix "secret/data/swarm/agents" s) && !(lib.hasInfix "secret/data/swarm/hives" s) @@ -938,18 +948,27 @@ let && !(noReaders ? swarm-bao-nats-callout-xkey); } { - # The admin password is created under Grafana's leaf, so its unit waits - # for the unit writing that leaf's role and policy; before it, the create - # is refused. The second arm is the absence control. - name = "grafana's admin password unit is ordered after grafana's policy unit"; + # The admin password and the controller's Grafana token are written under + # Grafana's leaf, so their units wait for the unit writing that leaf's + # role and policy; before it, the write is refused. The last arm is the + # absence control. + name = "grafana's admin password and controller token units are ordered after grafana's policy unit"; ok = - let - u = baoGrantWithConsumers.systemd.services.swarm-bao-grafana-admin; - in - lib.elem "swarm-bao-grafana-oidc-policy.service" u.after - && lib.elem "swarm-bao-grafana-oidc-policy.service" u.wants - && !(lib.elem "swarm-bao-grafana-oidc-policy.service" u.requires) - && !(baoGrantNoReaders.systemd.services ? swarm-bao-grafana-admin); + lib.all + ( + unit: + let + u = baoGrantWithConsumers.systemd.services.${unit}; + in + lib.elem "swarm-bao-grafana-oidc-policy.service" u.after + && lib.elem "swarm-bao-grafana-oidc-policy.service" u.wants + && !(lib.elem "swarm-bao-grafana-oidc-policy.service" u.requires) + && !(baoGrantNoReaders.systemd.services ? ${unit}) + ) + [ + "swarm-bao-grafana-admin" + "swarm-bao-grafana-controller-token" + ]; } { # The ordering is set apart from each reader's own definition, so it can @@ -1100,6 +1119,7 @@ let "swarm-bao-forwarder-oidc" "swarm-bao-grafana-admin" "swarm-bao-grafana-metrics-bearer" + "swarm-bao-grafana-controller-token" ] ++ policyReaders ); @@ -1715,6 +1735,25 @@ let "swarm-bao-forge-publisher-policy" ]; } + { + # The controller's Grafana token: read on the one leaf, pinned as the + # whole stanza so an added write fails, and named by no policy but the + # controller's and Grafana's, which writes it. + name = "the controller reads its Grafana token, and only Grafana's grant writes it"; + ok = + let + s = baoGrantHere.systemd.services.swarm-bao-controller-policy.script; + naming = lib.filter ( + u: lib.hasInfix "swarm-controller/grafana/token" baoGrantHere.systemd.services.${u}.script + ) grantingUnitNames; + in + lib.hasInfix "path \"secret/data/swarm/controller/swarm-controller/grafana/token\" {\n capabilities = [\"read\"]\n}" s + && + lib.sort lib.lessThan naming == [ + "swarm-bao-controller-policy" + "swarm-bao-grafana-oidc-policy" + ]; + } { # The forge's OIDC client secret by name: the forge publisher reads it, # and the secret publisher writes it under its `swarm/services/*` grant, diff --git a/nix/module-eval/grafana.nix b/nix/module-eval/grafana.nix index d9cbeebf..9f6e712e 100644 --- a/nix/module-eval/grafana.nix +++ b/nix/module-eval/grafana.nix @@ -249,6 +249,33 @@ let && lib.hasInfix "admin reset-admin-password --password-from-stdin < /var/lib/grafana-admin/password" service.preStart && lib.hasInfix "-config ${resetConfig}" service.script; } + { + # swarm-controller reads alerts with this token, so the account it + # belongs to is created and kept a Viewer, the token lands at the path + # the controller's grant reads, and an attempt against a Grafana or a + # store that never answers ends and retries. The timer re-runs it, so + # the unit must not stay active. + name = "the controller's Grafana token is a Viewer account's, stored under Grafana's leaf, and each attempt is bounded"; + ok = + let + unit = grafanaRemoteAuthelia.systemd.services.swarm-bao-grafana-controller-token; + s = unit.script; + in + !(lib.elem (toString (unit.serviceConfig.TimeoutStartSec or "infinity")) [ + "infinity" + "0" + ]) + && lib.hasInfix "curl --silent --show-error --connect-timeout 5 --max-time 15" s + && lib.hasInfix "--data '{\"name\":\"swarm-controller\",\"role\":\"Viewer\",\"isDisabled\":false}'" s + && lib.hasInfix "--data '{\"role\":\"Viewer\",\"isDisabled\":false}'" s + && lib.hasInfix "bao kv put secret/swarm/controller/swarm-controller/grafana/token value=-" s + && lib.hasInfix "--unix-socket /run/swarm-grafana/grafana.sock" s + && unit.environment.BAO_CLIENT_CERT == "/etc/pki/bao-grafana-oidc.pem" + && unit.serviceConfig.Restart == "on-failure" + && !(unit.serviceConfig.RemainAfterExit or false) + && lib.elem "swarm-bao-grafana-admin.service" unit.after + && grafanaRemoteAuthelia.systemd.timers ? swarm-bao-grafana-controller-token; + } { # Same 403-not-a-miss reason as the matrix and queue arms below: the # reader's grant covers the `services` prefix, so a path outside it is diff --git a/swarm-controller/README.md b/swarm-controller/README.md index dcbd560f..4bc39f23 100644 --- a/swarm-controller/README.md +++ b/swarm-controller/README.md @@ -73,6 +73,9 @@ The swarm's control plane. The swarm UI and `swarmctl` are its clients. [`docs/swarm/review-classifier.md`](../docs/swarm/review-classifier.md#swarm-level-settings). - **Relays** — each agent's terminal and turn-state header as SSE, agent icons, the cross-repo issue report, and the UI's quick links. +- **Alerts** — the swarm Grafana's current alerts (`GET /api/alerts`), read + at request time with a Viewer service-account token from the secret store + → [`docs/swarm/secrets.md`](../docs/swarm/secrets.md). It reads its configuration once at startup, from the environment the nix module sets. It persists `webhook-secret` and `operator-matrix-admins.json` in its diff --git a/swarm-controller/src/agent_limits.rs b/swarm-controller/src/agent_limits.rs index 9d7005a9..832bd21b 100644 --- a/swarm-controller/src/agent_limits.rs +++ b/swarm-controller/src/agent_limits.rs @@ -508,6 +508,7 @@ pub(crate) mod tests { reviews: Arc::default(), try_outs: crate::review_classifier::TryOuts::default(), runs: crate::review_classifier::runs::Runs::Live(None), + alerts: None, } } diff --git a/swarm-controller/src/alerts.rs b/swarm-controller/src/alerts.rs new file mode 100644 index 00000000..662b6387 --- /dev/null +++ b/swarm-controller/src/alerts.rs @@ -0,0 +1,391 @@ +//! `GET /api/alerts`: the alerts the swarm's Grafana holds right now, read at +//! request time for swarm-ui's alerts page. +//! +//! Grafana's URL comes from `swarm-controller.nix` +//! (`SWARM_CONTROLLER_GRAFANA_URL`). The token is a Viewer service account's, +//! read from the swarm's secret store at +//! `swarm_secret_client::grafana::controller_token_path` under this daemon's +//! own certificate. `swarm-bao-grafana-controller-token`, on Grafana's host, +//! mints it and replaces it whenever Grafana refuses the stored one, so a 401 +//! here re-reads the store once before giving up. + +use std::collections::BTreeMap; +use std::future::Future; +use std::pin::Pin; +use std::sync::Arc; +use std::time::Duration; + +use anyhow::{Context, Result}; +use axum::Json; +use axum::extract::State; +use axum::http::StatusCode; +use problem_details::ProblemDetails; +use serde::{Deserialize, Serialize}; +use utoipa::ToSchema; + +use crate::{AppState, error_problem}; + +/// The env var `swarm-controller.nix` sets. +const URL_ENV: &str = "SWARM_CONTROLLER_GRAFANA_URL"; + +/// Grafana's Prometheus-compatible list of alert instances, relative to its +/// base URL. Instances in the `Normal` state are not listed. +const ALERTS_PATH: &str = "api/prometheus/grafana/api/v1/alerts"; + +/// Bound on reaching Grafana. +const CONNECT_TIMEOUT: Duration = Duration::from_secs(5); +/// Bound on one whole request to Grafana, body included. +const TIMEOUT: Duration = Duration::from_secs(15); + +type TokenFuture = Pin> + Send>>; + +/// Where the token comes from: the store in the daemon, a stand-in in tests. +pub(crate) type TokenSource = Arc TokenFuture + Send + Sync>; + +/// One alert instance, as `GET /api/alerts` serves it. +#[derive(Clone, Debug, PartialEq, Serialize, ToSchema)] +pub(crate) struct Alert { + /// The rule's title (Grafana's `alertname` label). + pub name: String, + /// Grafana's state for the instance: `Alerting`, `Pending`, `NoData` or + /// `Error`. + pub state: String, + /// When the instance entered its state, RFC 3339, as Grafana reports it. + pub active_at: Option, + /// The rule's `summary` annotation, templated by Grafana. + pub summary: Option, + /// Every label of the instance, as Grafana reports them. + pub labels: BTreeMap, +} + +#[derive(Deserialize)] +struct PromResponse { + data: PromData, +} + +#[derive(Deserialize)] +struct PromData { + alerts: Vec, +} + +#[derive(Deserialize)] +struct PromAlert { + #[serde(default)] + labels: BTreeMap, + #[serde(default)] + annotations: BTreeMap, + state: String, + #[serde(rename = "activeAt")] + active_at: Option, +} + +impl From for Alert { + fn from(a: PromAlert) -> Self { + Self { + name: a.labels.get("alertname").cloned().unwrap_or_default(), + state: a.state, + active_at: a.active_at, + summary: a.annotations.get("summary").cloned(), + labels: a.labels, + } + } +} + +/// Grafana's alerts API, with the token last read from the store. +pub(crate) struct Grafana { + http: reqwest::Client, + alerts_url: url::Url, + source: TokenSource, + token: tokio::sync::Mutex>, +} + +impl Grafana { + /// # Errors + /// A base URL the alerts path cannot be joined onto, or a client that + /// cannot be built. + pub(crate) fn new(base: &url::Url, source: TokenSource, timeout: Duration) -> Result { + let http = reqwest::Client::builder() + .connect_timeout(CONNECT_TIMEOUT) + .timeout(timeout) + .build() + .context("building the Grafana HTTP client")?; + Ok(Self { + http, + alerts_url: base + .join(ALERTS_PATH) + .with_context(|| format!("joining {ALERTS_PATH} onto {base}"))?, + source, + token: tokio::sync::Mutex::new(None), + }) + } + + /// The held token, or a fresh one from the source when `reread` or none is + /// held yet. + async fn token(&self, reread: bool) -> Result { + let mut held = self.token.lock().await; + if let Some(t) = held.as_ref().filter(|_| !reread) { + return Ok(t.clone()); + } + let fresh = (self.source)().await?; + *held = Some(fresh.clone()); + Ok(fresh) + } + + async fn get(&self, token: &str) -> Result { + self.http + .get(self.alerts_url.clone()) + .bearer_auth(token) + .send() + .await + .with_context(|| format!("GET {}", self.alerts_url)) + } + + /// Every alert instance Grafana lists now. + /// + /// # Errors + /// No token in the store, Grafana unreachable or slower than the client's + /// timeout, a 401 for both the held token and a re-read one, any other + /// non-success status, or a body that is not the alerts shape. + pub(crate) async fn alerts(&self) -> Result> { + let mut res = self.get(&self.token(false).await?).await?; + if res.status() == reqwest::StatusCode::UNAUTHORIZED { + res = self.get(&self.token(true).await?).await?; + } + let status = res.status(); + if !status.is_success() { + anyhow::bail!("Grafana answered {status} to GET {}", self.alerts_url); + } + let body: PromResponse = res + .json() + .await + .with_context(|| format!("reading Grafana's answer to GET {}", self.alerts_url))?; + Ok(body.data.alerts.into_iter().map(Alert::from).collect()) + } +} + +/// One login and one read. An empty value is refused like an absent one. +async fn token_from_store() -> Result { + let path = swarm_secret_client::grafana::controller_token_path()?; + let store = crate::store::connect() + .await + .context("logging in to the swarm secret store")?; + let stored: Option = store + .read_optional(&path) + .await + .with_context(|| format!("reading {path}"))?; + match stored { + Some(t) if !t.value.trim().is_empty() => Ok(t.value.trim().to_owned()), + _ => anyhow::bail!( + "nothing at {path}: swarm-bao-grafana-controller-token, on the host that runs \ + Grafana, has not stored this daemon's Grafana token" + ), + } +} + +/// The daemon's Grafana client, or `None` when `SWARM_CONTROLLER_GRAFANA_URL` +/// is unset or unusable, which is logged. +pub(crate) fn from_env() -> Option> { + let raw = std::env::var(URL_ENV).ok()?; + let built = url::Url::parse(&raw) + .with_context(|| format!("parse {URL_ENV} ({raw}) as a URL")) + .and_then(|base| { + let source: TokenSource = Arc::new(|| Box::pin(token_from_store())); + Grafana::new(&base, source, TIMEOUT) + }); + match built { + Ok(g) => Some(Arc::new(g)), + Err(e) => { + tracing::warn!(error = %format!("{e:#}"), "Grafana misconfigured; /api/alerts is off"); + None + } + } +} + +/// The alerts the swarm's Grafana holds now: firing, pending, and those whose +/// rule has no data or failed to evaluate. +#[utoipa::path( + get, + path = "/api/alerts", + responses( + (status = 200, description = "every alert instance Grafana lists now", body = Vec), + (status = 503, description = "no Grafana is configured here, its token is not in the store \ + yet, or Grafana refused, failed or did not answer in time (problem+json)", body = String), + ), + tag = "alerts" +)] +pub(crate) async fn get_alerts( + State(state): State, +) -> Result>, ProblemDetails> { + let Some(grafana) = state.alerts.as_ref() else { + return Err(error_problem( + StatusCode::SERVICE_UNAVAILABLE, + "no Grafana is configured on this host", + )); + }; + grafana.alerts().await.map(Json).map_err(|e| { + let detail = format!("{e:#}"); + tracing::warn!(error = %detail, "reading Grafana's alerts failed"); + error_problem(StatusCode::SERVICE_UNAVAILABLE, &detail) + }) +} + +#[cfg(test)] +mod tests { + use super::*; + use axum::body::Body; + use axum::http::{HeaderMap, Request}; + use std::sync::Mutex; + use tower::ServiceExt as _; + + /// What the stand-in Grafana answers to a request with the accepted token. + const ALERTS_BODY: &str = r#"{"status":"success","data":{"alerts":[{ + "labels":{"alertname":"swarm queue connect failed","hive":"pr1ma","agent":"atlas"}, + "annotations":{"summary":"pr1ma/atlas could not connect"}, + "state":"Alerting","activeAt":"2026-10-15T09:00:00Z","value":"1e+00"}]}}"#; + + /// A stand-in Grafana on a loopback port: the alerts path answers + /// [`ALERTS_BODY`] to `Bearer ` and 401 to anything else, after + /// `delay`. + async fn fake_grafana(accepts: &'static str, delay: Duration) -> url::Url { + let app = axum::Router::new().route( + &format!("/{ALERTS_PATH}"), + axum::routing::get(move |headers: HeaderMap| async move { + tokio::time::sleep(delay).await; + let ok = headers + .get(axum::http::header::AUTHORIZATION) + .is_some_and(|v| v == format!("Bearer {accepts}").as_str()); + if ok { + (StatusCode::OK, ALERTS_BODY) + } else { + (StatusCode::UNAUTHORIZED, "") + } + }), + ); + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + tokio::spawn(async move { axum::serve(listener, app).await }); + url::Url::parse(&format!("http://{addr}/")).unwrap() + } + + /// A token source handing out `tokens` in order, the last one forever, + /// counting its calls. + fn tokens(tokens: &'static [&'static str]) -> (TokenSource, Arc>) { + let calls = Arc::new(Mutex::new(0)); + let counted = Arc::clone(&calls); + let source: TokenSource = Arc::new(move || { + let mut n = counted.lock().unwrap(); + let t = tokens[(*n).min(tokens.len() - 1)]; + *n += 1; + Box::pin(async move { Ok(t.to_owned()) }) + }); + (source, calls) + } + + async fn get_alerts_with(grafana: Option) -> (StatusCode, serde_json::Value) { + let (mut state, _) = crate::tests::state_with_roster(); + state.alerts = grafana.map(Arc::new); + let req = Request::get("/api/alerts") + .body(Body::empty()) + .expect("a request"); + let res = crate::build_app(state) + .oneshot(req) + .await + .expect("infallible"); + let status = res.status(); + let bytes = axum::body::to_bytes(res.into_body(), usize::MAX) + .await + .expect("a body"); + (status, serde_json::from_slice(&bytes).unwrap_or_default()) + } + + #[tokio::test] + async fn serves_grafanas_alerts_read_with_the_stored_token() { + let base = fake_grafana("viewer-token", Duration::ZERO).await; + let (source, calls) = tokens(&["viewer-token"]); + let grafana = Grafana::new(&base, source, TIMEOUT).unwrap(); + let (status, body) = get_alerts_with(Some(grafana)).await; + assert_eq!(status, StatusCode::OK, "{body}"); + assert_eq!( + body, + serde_json::json!([{ + "name": "swarm queue connect failed", + "state": "Alerting", + "active_at": "2026-10-15T09:00:00Z", + "summary": "pr1ma/atlas could not connect", + "labels": { + "alertname": "swarm queue connect failed", + "hive": "pr1ma", + "agent": "atlas", + }, + }]) + ); + assert_eq!(*calls.lock().unwrap(), 1); + } + + #[tokio::test] + async fn the_held_token_is_reused_across_requests() { + let base = fake_grafana("viewer-token", Duration::ZERO).await; + let (source, calls) = tokens(&["viewer-token"]); + let grafana = Grafana::new(&base, source, TIMEOUT).unwrap(); + grafana.alerts().await.unwrap(); + grafana.alerts().await.unwrap(); + assert_eq!(*calls.lock().unwrap(), 1); + } + + /// The minting unit replaces a token Grafana refuses, so a 401 re-reads + /// the store and sends the request once more. + #[tokio::test] + async fn a_refused_token_is_read_again_from_the_store() { + let base = fake_grafana("replaced", Duration::ZERO).await; + let (source, calls) = tokens(&["stale", "replaced"]); + let grafana = Grafana::new(&base, source, TIMEOUT).unwrap(); + let alerts = grafana.alerts().await.unwrap(); + assert_eq!(alerts.len(), 1); + assert_eq!(*calls.lock().unwrap(), 2); + } + + #[tokio::test] + async fn a_token_refused_twice_is_a_503_naming_the_status() { + let base = fake_grafana("replaced", Duration::ZERO).await; + let (source, calls) = tokens(&["stale"]); + let grafana = Grafana::new(&base, source, TIMEOUT).unwrap(); + let (status, body) = get_alerts_with(Some(grafana)).await; + assert_eq!(status, StatusCode::SERVICE_UNAVAILABLE); + assert!(body["detail"].as_str().unwrap().contains("401"), "{body}"); + assert_eq!(*calls.lock().unwrap(), 2); + } + + #[tokio::test] + async fn a_grafana_slower_than_the_timeout_is_a_503() { + let base = fake_grafana("viewer-token", Duration::from_secs(5)).await; + let (source, _) = tokens(&["viewer-token"]); + let grafana = Grafana::new(&base, source, Duration::from_millis(200)).unwrap(); + let started = std::time::Instant::now(); + let (status, _) = get_alerts_with(Some(grafana)).await; + assert_eq!(status, StatusCode::SERVICE_UNAVAILABLE); + assert!(started.elapsed() < Duration::from_secs(4)); + } + + #[tokio::test] + async fn no_token_in_the_store_is_a_503_saying_so() { + let base = fake_grafana("viewer-token", Duration::ZERO).await; + let source: TokenSource = + Arc::new(|| Box::pin(async { anyhow::bail!("nothing at the token path") })); + let grafana = Grafana::new(&base, source, TIMEOUT).unwrap(); + let (status, body) = get_alerts_with(Some(grafana)).await; + assert_eq!(status, StatusCode::SERVICE_UNAVAILABLE); + assert!( + body["detail"] + .as_str() + .unwrap() + .contains("nothing at the token path"), + "{body}" + ); + } + + #[tokio::test] + async fn no_grafana_configured_is_a_503() { + let (status, _) = get_alerts_with(None).await; + assert_eq!(status, StatusCode::SERVICE_UNAVAILABLE); + } +} diff --git a/swarm-controller/src/forge_account.rs b/swarm-controller/src/forge_account.rs index fb379ba3..aa3d19f3 100644 --- a/swarm-controller/src/forge_account.rs +++ b/swarm-controller/src/forge_account.rs @@ -168,6 +168,7 @@ mod tests { try_outs: super::super::review_classifier::TryOuts::default(), reviews: std::sync::Arc::default(), runs: crate::review_classifier::runs::Runs::Live(None), + alerts: None, } } diff --git a/swarm-controller/src/github_account.rs b/swarm-controller/src/github_account.rs index b54e8b1e..794cb370 100644 --- a/swarm-controller/src/github_account.rs +++ b/swarm-controller/src/github_account.rs @@ -138,6 +138,7 @@ mod tests { try_outs: super::super::review_classifier::TryOuts::default(), reviews: std::sync::Arc::default(), runs: crate::review_classifier::runs::Runs::Live(None), + alerts: None, } } diff --git a/swarm-controller/src/linked_accounts.rs b/swarm-controller/src/linked_accounts.rs index 7ec5905b..c454b73d 100644 --- a/swarm-controller/src/linked_accounts.rs +++ b/swarm-controller/src/linked_accounts.rs @@ -973,6 +973,7 @@ pub(crate) mod tests { try_outs: super::super::review_classifier::TryOuts::default(), reviews: std::sync::Arc::default(), runs: crate::review_classifier::runs::Runs::Live(None), + alerts: None, } } diff --git a/swarm-controller/src/main.rs b/swarm-controller/src/main.rs index 52952535..dd30e4db 100644 --- a/swarm-controller/src/main.rs +++ b/swarm-controller/src/main.rs @@ -50,6 +50,7 @@ mod agent_login; mod agent_renewal; mod agent_state_stream; mod agent_status; +mod alerts; mod auth; mod builder; mod claude_login; @@ -839,6 +840,7 @@ fn socket_path() -> PathBuf { (name = "schedules", description = "scheduled prompts (`swarmctl schedule`)"), (name = "review-classifier", description = "trying review classifications against a PR (swarm-ui's try-out page), and the classifier's swarm-wide settings"), (name = "pulls", description = "open pull requests, for the try-out page's dropdowns"), + (name = "alerts", description = "the swarm Grafana's current alerts (swarm-ui's alerts page)"), ) )] struct ApiDoc; @@ -972,6 +974,9 @@ struct AppState { reviews: Arc, /// The review classifier's finished runs. runs: review_classifier::runs::Runs, + /// Grafana's alerts API, for `GET /api/alerts`. `None` when no Grafana URL + /// is configured. + alerts: Option>, } /// Env var the controller's NixOS module sets from @@ -3451,6 +3456,7 @@ async fn main() -> Result<()> { try_outs: review_classifier::TryOuts::default(), reviews: Arc::default(), runs, + alerts: alerts::from_env(), }; let app = build_app(state); @@ -3468,6 +3474,7 @@ async fn main() -> Result<()> { fn build_app(state: AppState) -> axum::Router { let (router, api) = OpenApiRouter::::with_openapi(ApiDoc::openapi()) .routes(routes!(health)) + .routes(routes!(alerts::get_alerts)) .routes(routes!(get_hives)) .routes(routes!(get_hives_status)) .routes(routes!(get_links)) @@ -3660,6 +3667,7 @@ mod tests { try_outs: super::review_classifier::TryOuts::default(), reviews: std::sync::Arc::default(), runs: super::review_classifier::runs::Runs::Live(None), + alerts: None, }; (state, sched) } diff --git a/swarm-controller/src/matrix_account.rs b/swarm-controller/src/matrix_account.rs index 571b9189..3b469411 100644 --- a/swarm-controller/src/matrix_account.rs +++ b/swarm-controller/src/matrix_account.rs @@ -585,6 +585,7 @@ mod tests { try_outs: super::super::review_classifier::TryOuts::default(), reviews: std::sync::Arc::default(), runs: crate::review_classifier::runs::Runs::Live(None), + alerts: None, } } diff --git a/swarm-secret-client/src/grafana.rs b/swarm-secret-client/src/grafana.rs new file mode 100644 index 00000000..b1775e2c --- /dev/null +++ b/swarm-secret-client/src/grafana.rs @@ -0,0 +1,54 @@ +//! The Grafana agreement: where `swarm-controller`'s token for Grafana's API +//! lives, and what the object at that path holds. +//! +//! `swarm-bao-grafana-controller-token`, on the host that runs Grafana, mints +//! the token for a Viewer service account and writes it here under Grafana's +//! own certificate; `swarm-controller` reads it under its own. Those two +//! principals' stanzas are the only ones in the store that cover this path. + +use serde::{Deserialize, Serialize}; + +use crate::{ + Error, + path::{Kind, principal_prefix}, +}; + +/// The path holding `swarm-controller`'s Grafana token. +/// +/// Under [`Kind::Controller`], which no hive's policy reads. The nix half is +/// `swarm-bao.nix`'s `controllerGrafanaTokenLeaf`. +/// +/// # Errors +/// Never in practice: the name segment is a constant. The `Result` is +/// [`principal_prefix`]'s. +pub fn controller_token_path() -> Result { + let prefix = principal_prefix(Kind::Controller, crate::matrix::CONTROLLER)?; + Ok(format!("{prefix}/grafana/token")) +} + +/// What [`controller_token_path`] holds. +/// +/// No `Debug` derive: `value` reads every alert and dashboard in the swarm's +/// Grafana, and a derived `Debug` is one `{:?}` away from a log line. +#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct ControllerToken { + /// The token itself. The minting unit writes it with + /// `bao kv put … value=-`, so this name is load-bearing for a writer this + /// crate does not control. + pub value: String, +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn the_controllers_grafana_token_is_under_the_controllers_prefix() { + // The literal is what `swarm-bao.nix` grants Grafana's role and the + // controller's, and what the minting unit writes. + assert_eq!( + controller_token_path().expect("a constant segment is legal"), + "swarm/controller/swarm-controller/grafana/token" + ); + } +} diff --git a/swarm-secret-client/src/lib.rs b/swarm-secret-client/src/lib.rs index 229dda4b..7ee55c28 100644 --- a/swarm-secret-client/src/lib.rs +++ b/swarm-secret-client/src/lib.rs @@ -5,10 +5,10 @@ //! the rules every path obeys ([`path`]), the translation from this //! deployment's environment into a logged-in client ([`client`]), and, per kind //! of secret, the path it lives at together with the fields it holds -//! ([`matrix`], [`queue`], [`mtls`], [`forge`], [`github`], [`acp`], [`ci`], -//! [`review_classifier`], [`builder`], [`claude_login`]). Each of those is a -//! thing the controller and a hive must say identically, so it is said once -//! here. +//! ([`matrix`], [`queue`], [`mtls`], [`forge`], [`github`], [`grafana`], +//! [`acp`], [`ci`], [`review_classifier`], [`builder`], [`claude_login`]). +//! Each of those is a thing the controller and a hive must say identically, so +//! it is said once here. //! //! [`policy`] is the same kind of agreement seen from the other side: which of //! those paths a given principal's own token may read — a hive's, and an @@ -31,6 +31,7 @@ pub mod claude_login; pub mod client; pub mod forge; pub mod github; +pub mod grafana; pub mod matrix; pub mod mtls; pub mod path;