swarm UI: an alerts page, read by the controller with a Viewer Grafana token from bao
swarm-bao-grafana-controller-token, on the host that runs Grafana, logs in to Grafana's API over its socket as the built-in admin (the password swarm-bao-grafana-admin minted, #5138) and keeps a `swarm-controller` service account that is an enabled Viewer. It mints a token for that account and writes it to bao at swarm/controller/swarm-controller/grafana/token under Grafana's leaf. A ten-minute timer runs it again. The unit keeps a stored token Grafana accepts. If Grafana refuses the stored token with 401, or bao holds none, the unit deletes every token of the account and mints a new one. Grafana's grant gains create/read/update on that leaf, and the controller's gains read. swarm-controller serves GET /api/alerts from Grafana's /api/prometheus/grafana/api/v1/alerts. It reads the token from bao, keeps it in memory, and re-reads bao once on a 401. The HTTP client has 5s connect and 15s request timeouts. Any failure is a 503 that names its cause. swarm-ui gets an /alerts page listing them, built the same way as the hives page. This is option (c) on #5129, the operator's choice.
This commit is contained in:
atlas
mara
parent
8cb6273782
commit
7d09776742
23 changed files with 888 additions and 37 deletions
|
|
@ -60,6 +60,7 @@ neither is a renaming of the other.
|
|||
| a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) |
|
||||
| the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf** | `/var/lib/hive-otel-oidc/hive-<name>.secret` on the hive's host, 0400 root, loaded by `LoadCredential` | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) |
|
||||
| the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | `/var/lib/forgejo/swarm-controller-token` inside the forge container, the minter's record that it has minted; `swarm-bao-forge-publisher` copies it into the swarm secret store at `swarm/controller/swarm-controller/forge/token` each time the forge container starts and each time the mint writes the file; the controller holds it in memory only | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) |
|
||||
| the swarm controller's Grafana token | `swarm-bao-grafana-controller-token.service`, **on the host that runs Grafana**, as Grafana's built-in admin over its socket: it creates the `swarm-controller` service account as a Viewer (and sets it back to Viewer on every run), mints a token for it and writes it to the swarm secret store at `swarm/controller/swarm-controller/grafana/token` under Grafana's leaf. Re-run every ten minutes: it keeps a stored token Grafana accepts, and replaces one Grafana refuses with 401 (or a missing one), deleting every other token of that account first | the swarm secret store only; the controller reads it for `GET /api/alerts` and holds it in memory, re-reading the store once when Grafana answers 401 | nothing to place: Grafana's leaf is already required to run Grafana. Until the unit has run, `/api/alerts` answers 503 naming the empty path |
|
||||
| an agent's claude login (`.credentials.json`) | `swarm-claude-login.service`, in its container, when an operator completes `claude auth login` from the agent's card in the swarm UI; it stores the file as a new version at `swarm/agents/<agent>/claude-credentials` under its own leaf, which may create and update that path and read nothing under an agent | the swarm secret store at that path; the agent's `hive-agent` reads it under the agent's own leaf every 30 seconds while it waits for a login and writes it to `~/.claude/.credentials.json` | the same read, co-located or not; nothing for an operator to place beyond the agent's store leaf. When the store's copy and the agent's differ the KV version decides: the agent installs only a version newer than the last it installed (`claude-credentials-version` under the harness dir), because claude rotates the refresh token in its own copy and the stored one goes stale. The agent ignores a stale or equal version, and a relogin writes a newer one — [claude-login.md](claude-login.md) |
|
||||
| the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
|
||||
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
|
||||
|
|
|
|||
|
|
@ -7,17 +7,18 @@ domain, covers host-level detail for one hive.
|
|||
|
||||
## What it shows
|
||||
|
||||
| route | what |
|
||||
| ---------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `/` | the start page: the swarm's name over a digital-rain backdrop; the swarm name in the header links here |
|
||||
| `/hives` | the hive directory, each hive with its last reported status |
|
||||
| `/agents` | every agent: status, config PR, wanted state, inference settings, resource caps; create agents, link forge, matrix and GitHub accounts |
|
||||
| `/agents/<name>/terminal` | one agent's live terminal |
|
||||
| `/agents/<name>/subagents/<subagent>/terminal` | one of that agent's subagents' live terminal |
|
||||
| `/jobs` | the controller's job graph — where agent creation and credential mints show progress |
|
||||
| `/schedules` | the scheduled prompts: add, edit, pause, resume and remove |
|
||||
| `/issues` | a cross-repo issue report |
|
||||
| `/review` | the review classifier: its recent runs, a try-out on a real PR, and its settings |
|
||||
| route | what |
|
||||
| ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `/` | the start page: the swarm's name over a digital-rain backdrop; the swarm name in the header links here |
|
||||
| `/hives` | the hive directory, each hive with its last reported status |
|
||||
| `/agents` | every agent: status, config PR, wanted state, inference settings, resource caps; create agents, link forge, matrix and GitHub accounts |
|
||||
| `/agents/<name>/terminal` | one agent's live terminal |
|
||||
| `/agents/<name>/subagents/<subagent>/terminal` | one of that agent's subagents' live terminal |
|
||||
| `/alerts` | the alerts the swarm's Grafana holds now (firing, pending, no data, error), read through the controller with a Viewer service-account token |
|
||||
| `/jobs` | the controller's job graph — where agent creation and credential mints show progress |
|
||||
| `/schedules` | the scheduled prompts: add, edit, pause, resume and remove |
|
||||
| `/issues` | a cross-repo issue report |
|
||||
| `/review` | the review classifier: its recent runs, a try-out on a real PR, and its settings |
|
||||
|
||||
Everything it shows comes from [`swarm-controller`](../../swarm-controller/README.md).
|
||||
An agent created here or with `swarmctl agent create` starts `paused`; set it
|
||||
|
|
|
|||
|
|
@ -4,6 +4,7 @@ import { Route, Switch } from "wouter-preact";
|
|||
import { Shell } from "./shell/Shell.js";
|
||||
import { AgentsPage } from "./pages/agents/AgentsPage.js";
|
||||
import { AgentTerminalPage } from "./pages/agents/AgentTerminalPage.js";
|
||||
import { AlertsPage } from "./pages/AlertsPage.js";
|
||||
import { ComponentsPage } from "./pages/ComponentsPage.js";
|
||||
import { JobsPage } from "./pages/JobsPage.js";
|
||||
import { HivesPage } from "./pages/HivesPage.js";
|
||||
|
|
@ -27,6 +28,7 @@ export function App() {
|
|||
<Switch>
|
||||
<Route path="/" component={StartPage} />
|
||||
<Route path="/hives" component={HivesPage} />
|
||||
<Route path="/alerts" component={AlertsPage} />
|
||||
<Route path="/agents" component={AgentsPage} />
|
||||
<Route path="/agents/:name/terminal" component={AgentTerminalPage} />
|
||||
<Route
|
||||
|
|
|
|||
138
frontend/packages/swarm-ui/src/pages/AlertsPage.tsx
Normal file
138
frontend/packages/swarm-ui/src/pages/AlertsPage.tsx
Normal file
|
|
@ -0,0 +1,138 @@
|
|||
// <AlertsPage> — `/alerts`' content: the alerts the swarm's Grafana holds
|
||||
// now, from swarm-controller's `GET /api/alerts`. Same shape as `HivesPage`:
|
||||
// one fetch on a `RefreshIntervalPicker` cadence, rendered as a `Table`.
|
||||
import { useState } from "preact/hooks";
|
||||
import { ApiErrorPanel } from "@hive/shared/api-error-panel.js";
|
||||
import { readApiError, type ProblemDetails } from "@hive/shared/api-error.js";
|
||||
import { Badge, type BadgeTone } from "@hive/shared/badge.js";
|
||||
import { Panel } from "../ui/panel/Panel.js";
|
||||
import { RelativeTime } from "../ui/relative-time/RelativeTime.js";
|
||||
import {
|
||||
RefreshIntervalPicker,
|
||||
useRefreshInterval,
|
||||
type RefreshIntervalMs,
|
||||
} from "../ui/refresh-interval/RefreshInterval.js";
|
||||
import { Table, type TableColumn } from "../ui/table/Table.js";
|
||||
|
||||
interface Alert {
|
||||
name: string;
|
||||
// Grafana's state for the instance: `Alerting`, `Pending`, `NoData` or
|
||||
// `Error`. Anything else renders neutral.
|
||||
state: string;
|
||||
active_at: string | null;
|
||||
summary: string | null;
|
||||
labels: Record<string, string>;
|
||||
}
|
||||
|
||||
const STATE_TONE: Record<string, BadgeTone> = {
|
||||
Alerting: "negative",
|
||||
Error: "negative",
|
||||
NoData: "warning",
|
||||
Pending: "warning",
|
||||
};
|
||||
|
||||
// Grafana reports an instance that never became active with the zero time,
|
||||
// which `Date.parse` reads as year 1.
|
||||
function activeSinceMs(a: Alert): number | null {
|
||||
if (a.active_at === null) return null;
|
||||
const ms = Date.parse(a.active_at);
|
||||
return Number.isNaN(ms) || ms <= 0 ? null : ms;
|
||||
}
|
||||
|
||||
// A row needs a key that is unique among instances of the same rule.
|
||||
function alertKey(a: Alert): string {
|
||||
return JSON.stringify(Object.entries(a.labels).sort());
|
||||
}
|
||||
|
||||
const COLUMNS: TableColumn<Alert>[] = [
|
||||
{
|
||||
key: "name",
|
||||
header: "alert",
|
||||
render: (a) => a.name,
|
||||
sortBy: (a) => a.name,
|
||||
filterValue: (a) => a.name,
|
||||
},
|
||||
{
|
||||
key: "state",
|
||||
header: "state",
|
||||
render: (a) => {
|
||||
const since = activeSinceMs(a);
|
||||
return (
|
||||
<Badge
|
||||
tone={STATE_TONE[a.state] ?? "neutral"}
|
||||
value={
|
||||
since !== null ? (
|
||||
<>
|
||||
{a.state} (<RelativeTime epochMs={since} />)
|
||||
</>
|
||||
) : (
|
||||
a.state
|
||||
)
|
||||
}
|
||||
/>
|
||||
);
|
||||
},
|
||||
sortBy: (a) => a.state,
|
||||
filterValues: (a) => [a.state],
|
||||
filterMode: "multiselect",
|
||||
},
|
||||
{
|
||||
key: "summary",
|
||||
header: "summary",
|
||||
render: (a) => a.summary ?? "—",
|
||||
sortBy: (a) => a.summary ?? "",
|
||||
filterValue: (a) => a.summary ?? "",
|
||||
},
|
||||
];
|
||||
|
||||
const DEFAULT_INTERVAL_MS: RefreshIntervalMs = 30_000;
|
||||
|
||||
export function AlertsPage() {
|
||||
const [alerts, setAlerts] = useState<Alert[] | null>(null);
|
||||
const [error, setError] = useState<ProblemDetails | null>(null);
|
||||
const [intervalMs, setIntervalMs] =
|
||||
useState<RefreshIntervalMs>(DEFAULT_INTERVAL_MS);
|
||||
|
||||
useRefreshInterval(intervalMs, () => {
|
||||
(async () => {
|
||||
const r = await fetch("/api/alerts");
|
||||
if (!r.ok) {
|
||||
setError(await readApiError(r));
|
||||
return;
|
||||
}
|
||||
setAlerts((await r.json()) as Alert[]);
|
||||
setError(null);
|
||||
})().catch((e: unknown) => setError({ detail: String(e) }));
|
||||
});
|
||||
|
||||
return (
|
||||
<Panel
|
||||
title="alerts"
|
||||
icon="🚨"
|
||||
actions={
|
||||
<RefreshIntervalPicker
|
||||
id="alerts-refresh"
|
||||
value={intervalMs}
|
||||
onChange={setIntervalMs}
|
||||
/>
|
||||
}
|
||||
>
|
||||
{error ? (
|
||||
<ApiErrorPanel
|
||||
context="failed to load Grafana's alerts"
|
||||
problem={error}
|
||||
/>
|
||||
) : null}
|
||||
{!error && alerts === null ? <p>loading…</p> : null}
|
||||
{alerts && alerts.length === 0 ? <p>no alerts.</p> : null}
|
||||
{alerts && alerts.length > 0 ? (
|
||||
<Table
|
||||
columns={COLUMNS}
|
||||
rows={alerts}
|
||||
rowKey={alertKey}
|
||||
storageKey="swarm-ui:alerts:table-filters"
|
||||
/>
|
||||
) : null}
|
||||
</Panel>
|
||||
);
|
||||
}
|
||||
|
|
@ -51,6 +51,7 @@ const MOTION_KEY = "swarm-ui:motion-override";
|
|||
const NAV_ITEMS: { href: string; label: string; accent: string }[] = [
|
||||
{ href: "/hives", label: "hives", accent: "var(--purple)" },
|
||||
{ href: "/agents", label: "agents", accent: "var(--green)" },
|
||||
{ href: "/alerts", label: "alerts", accent: "var(--red)" },
|
||||
{ href: "/jobs", label: "jobs", accent: "var(--pink)" },
|
||||
{ href: "/schedules", label: "schedules", accent: "var(--cyan)" },
|
||||
{ href: "/issues", label: "issues", accent: "var(--yellow)" },
|
||||
|
|
|
|||
|
|
@ -67,7 +67,8 @@ let
|
|||
];
|
||||
};
|
||||
# ./swarm-grafana.nix, under Grafana's leaf, which may create the admin
|
||||
# password and the metrics bearer only once that policy is written.
|
||||
# password, the metrics bearer and the controller's Grafana token only
|
||||
# once that policy is written.
|
||||
swarm-bao-grafana-admin = {
|
||||
here = deployCfg.grafana.enable;
|
||||
policies = [ "swarm-bao-grafana-oidc-policy" ];
|
||||
|
|
@ -76,6 +77,10 @@ let
|
|||
here = deployCfg.grafana.enable;
|
||||
policies = [ "swarm-bao-grafana-oidc-policy" ];
|
||||
};
|
||||
swarm-bao-grafana-controller-token = {
|
||||
here = deployCfg.grafana.enable;
|
||||
policies = [ "swarm-bao-grafana-oidc-policy" ];
|
||||
};
|
||||
# ./hive-matrix.nix, under matrix-ctl's leaf.
|
||||
swarm-bao-matrix-oidc = {
|
||||
here = deployCfg.matrix.enable;
|
||||
|
|
|
|||
|
|
@ -47,6 +47,7 @@ let
|
|||
swarm-bao-grafana-oidc = deployCfg.grafana.enable;
|
||||
swarm-bao-grafana-admin = deployCfg.grafana.enable;
|
||||
swarm-bao-grafana-metrics-bearer = deployCfg.grafana.enable;
|
||||
swarm-bao-grafana-controller-token = deployCfg.grafana.enable;
|
||||
# ./swarm-nats.nix
|
||||
swarm-bao-nats-callout-xkey = deployCfg.nats.enable;
|
||||
# ./swarm-ci-runner.nix
|
||||
|
|
|
|||
|
|
@ -369,9 +369,9 @@ let
|
|||
# Each hive's matrix sender token (`matrix_account::hive_sender`) grants `read`
|
||||
# for the same keep-if-live reason, and `+` for the same glob-only-as-last-segment reason.
|
||||
#
|
||||
# Three leaves it reads and never writes: the swarm appservice token
|
||||
# (matrix-ctl, `matrixCtlPolicyText` below), its own OIDC client secret (the
|
||||
# publisher) and its own forge token (the forge's host, `forgePublisherWriters`).
|
||||
# Four leaves it reads and never writes: the swarm appservice token (matrix-ctl),
|
||||
# its OIDC client secret (the publisher), its forge token (the forge's host)
|
||||
# and its Grafana token (Grafana's host). Each writer's grant is named below.
|
||||
#
|
||||
# The agent PKI grant is its only path on that mount: it can ask the one role
|
||||
# for a certificate, not write that role or reach the issuer.
|
||||
|
|
@ -457,6 +457,10 @@ let
|
|||
capabilities = ["read"]
|
||||
}
|
||||
|
||||
path "${credentialMountPath}/data/${controllerGrafanaTokenLeaf}" {
|
||||
capabilities = ["read"]
|
||||
}
|
||||
|
||||
path "${agentPkiMountPath}/issue/${agentPkiRoleName}" {
|
||||
capabilities = ["update"]
|
||||
}
|
||||
|
|
@ -582,6 +586,11 @@ let
|
|||
# it; the controller reads it and holds no other copy.
|
||||
controllerForgeTokenLeaf = "swarm/controller/swarm-controller/forge/token";
|
||||
|
||||
# The controller's Viewer token for Grafana's API, the nix half of
|
||||
# `swarm_secret_client::grafana::controller_token_path`. Grafana's host
|
||||
# writes it; the controller reads it and holds no other copy.
|
||||
controllerGrafanaTokenLeaf = "swarm/controller/swarm-controller/grafana/token";
|
||||
|
||||
# The forge's runner registration token, the nix half of
|
||||
# `swarm_secret_client::ci::registration_token_path`. The controller writes
|
||||
# it and cannot read it back (`forge::runner_token` keeps the value it last
|
||||
|
|
@ -911,6 +920,10 @@ let
|
|||
# in ./swarm-grafana.nix) and the bearer its metrics datasource presents
|
||||
# (`swarm-bao-grafana-metrics-bearer`). `create` without `update`, so a stored
|
||||
# value is never replaced.
|
||||
#
|
||||
# The controller's Grafana token is the one it may replace:
|
||||
# `swarm-bao-grafana-controller-token` mints a new one when Grafana refuses
|
||||
# the stored one, as it does after its database is recreated.
|
||||
grafanaOidcReaders = [
|
||||
{
|
||||
name = "swarm-grafana-oidc";
|
||||
|
|
@ -918,7 +931,12 @@ let
|
|||
policyText =
|
||||
readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.grafana.oidc.clientId}/oidc/client"
|
||||
+ createStanza "${credentialMountPath}/data/${grafanaAdminLeaf}"
|
||||
+ createStanza "${credentialMountPath}/data/${grafanaMetricsBearerLeaf}";
|
||||
+ createStanza "${credentialMountPath}/data/${grafanaMetricsBearerLeaf}"
|
||||
+ ''
|
||||
path "${credentialMountPath}/data/${controllerGrafanaTokenLeaf}" {
|
||||
capabilities = ["create", "read", "update"]
|
||||
}
|
||||
'';
|
||||
}
|
||||
];
|
||||
|
||||
|
|
|
|||
|
|
@ -811,6 +811,11 @@ in
|
|||
# separated: a matrix user id cannot contain a space. Consumed by
|
||||
# swarm-controller/src/matrix_account/operator_admin.rs.
|
||||
SWARM_CONTROLLER_MATRIX_ADMIN_SYNC_IGNORE_USERS = lib.concatStringsSep " " matrixCfg.adminSync.ignoreUsers;
|
||||
# Where `GET /api/alerts` reads alerts (swarm-controller/src/alerts.rs).
|
||||
# Every swarm has one Grafana, reached by name wherever it runs. The
|
||||
# token is in the store, written by `swarm-bao-grafana-controller-token`
|
||||
# on Grafana's host.
|
||||
SWARM_CONTROLLER_GRAFANA_URL = "https://${swarmCfg.grafana.domain}";
|
||||
}
|
||||
// queueEnv
|
||||
// forgeEnv
|
||||
|
|
|
|||
|
|
@ -273,6 +273,14 @@ let
|
|||
metricsBearerPath = "/var/lib/grafana-metrics/bearer";
|
||||
hostMetricsBearerPath = "/var/lib/nixos-containers/${cfg.machine}${metricsBearerPath}";
|
||||
|
||||
# swarm-controller's token for Grafana's API, minted by
|
||||
# `swarm-bao-grafana-controller-token` below for a Viewer service account of
|
||||
# this name. `controllerGrafanaTokenLeaf` in ./swarm-bao.nix is the nix half
|
||||
# of the path, `swarm_secret_client::grafana::controller_token_path` the Rust
|
||||
# half.
|
||||
controllerAccountName = "swarm-controller";
|
||||
controllerTokenStorePath = "secret/swarm/controller/swarm-controller/grafana/token";
|
||||
|
||||
socketPath = "${deployCfg.grafana.socketDir}/grafana.sock";
|
||||
|
||||
# Both static NixOS ids, and both checked rather than assumed: nginx has
|
||||
|
|
@ -849,6 +857,148 @@ in
|
|||
'';
|
||||
};
|
||||
|
||||
# swarm-controller reads Grafana's alerts with a token of a Viewer service
|
||||
# account, created here through Grafana's API as the built-in admin and
|
||||
# handed over through the store.
|
||||
#
|
||||
# Re-run by the timer below. A stored token Grafana still accepts is kept;
|
||||
# one it refuses with 401, or none at all, is replaced: every token of the
|
||||
# account is deleted and one new one is minted and stored. The account's
|
||||
# role is set back to Viewer on every run.
|
||||
systemd.services.swarm-bao-grafana-controller-token = {
|
||||
description = "give swarm-controller a Viewer token for Grafana's API through the swarm secret store";
|
||||
# Ordered after the store where it runs here:
|
||||
# ./glue-bao-readers-store-order.nix.
|
||||
after = [
|
||||
"container@${cfg.machine}.service"
|
||||
"swarm-bao-grafana-admin.service"
|
||||
];
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
path = [
|
||||
baoDeploy.package
|
||||
pkgs.coreutils
|
||||
pkgs.curl
|
||||
pkgs.jq
|
||||
];
|
||||
# ./lib/store-retry.nix. A Grafana still starting refuses the
|
||||
# connection, and that attempt fails into the same retry.
|
||||
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
|
||||
serviceConfig = storeRetry.serviceConfig // {
|
||||
# No `RemainAfterExit`: the timer below starts this again.
|
||||
Type = "oneshot";
|
||||
SyslogIdentifier = "swarm-bao-grafana-controller-token";
|
||||
# Grafana's answers, the new token among them, pass through a temp file.
|
||||
PrivateTmp = true;
|
||||
# Each call below is bounded by curl's own timeouts; a run makes up to
|
||||
# about ten of them.
|
||||
TimeoutStartSec = "3min";
|
||||
};
|
||||
environment = {
|
||||
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
|
||||
BAO_CLIENT_CERT = baoDeploy.grafanaOidcClientCertFile;
|
||||
BAO_CLIENT_KEY = baoDeploy.grafanaOidcClientKeyFile;
|
||||
}
|
||||
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
|
||||
BAO_CACERT = baoDeploy.serverCaFile;
|
||||
};
|
||||
script = ''
|
||||
set -euo pipefail
|
||||
|
||||
err="$(mktemp)"
|
||||
body="$(mktemp)"
|
||||
trap 'rm -f "$err" "$body"' EXIT
|
||||
|
||||
# One call to Grafana's API through its socket. Prints the HTTP
|
||||
# status; the body lands in "$body". The credential reaches curl as
|
||||
# a header file, never as an argument.
|
||||
grafana() {
|
||||
local auth="$1" method="$2" path="$3"
|
||||
shift 3
|
||||
curl --silent --show-error --connect-timeout 5 --max-time 15 \
|
||||
--unix-socket ${lib.escapeShellArg socketPath} \
|
||||
-H @<(printf 'Authorization: %s\n' "$auth") \
|
||||
-H 'Content-Type: application/json' \
|
||||
-X "$method" -o "$body" -w '%{http_code}' "$@" \
|
||||
"http://localhost$path"
|
||||
}
|
||||
|
||||
# Fails the attempt unless the last call answered one of the statuses
|
||||
# given.
|
||||
expect() {
|
||||
local what="$1" status="$2"
|
||||
shift 2
|
||||
for ok in "$@"; do
|
||||
[ "$status" = "$ok" ] && return 0
|
||||
done
|
||||
echo "Grafana answered $status to $what:" >&2
|
||||
cat "$body" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
if [ ! -s ${lib.escapeShellArg hostAdminPasswordPath} ]; then
|
||||
echo "no Grafana admin password at ${hostAdminPasswordPath} yet; swarm-bao-grafana-admin writes it." >&2
|
||||
exit 1
|
||||
fi
|
||||
admin="Basic $(printf 'admin:%s' "$(< ${lib.escapeShellArg hostAdminPasswordPath})" | base64 -w0)"
|
||||
|
||||
status="$(grafana "$admin" GET '/api/serviceaccounts/search?perpage=1000&query=${controllerAccountName}')"
|
||||
expect "the service account search" "$status" 200
|
||||
account="$(jq -c --arg n ${controllerAccountName} '[.serviceAccounts[] | select(.name == $n)][0] // empty' "$body")"
|
||||
if [ -z "$account" ]; then
|
||||
status="$(grafana "$admin" POST /api/serviceaccounts \
|
||||
--data '{"name":"${controllerAccountName}","role":"Viewer","isDisabled":false}')"
|
||||
expect "creating the ${controllerAccountName} service account" "$status" 200 201
|
||||
account="$(jq -c . "$body")"
|
||||
echo "created the ${controllerAccountName} service account as a Viewer"
|
||||
fi
|
||||
id="$(jq -r .id <<<"$account")"
|
||||
if [ "$(jq -r '"\(.role) \(.isDisabled // false)"' <<<"$account")" != "Viewer false" ]; then
|
||||
status="$(grafana "$admin" PATCH "/api/serviceaccounts/$id" \
|
||||
--data '{"role":"Viewer","isDisabled":false}')"
|
||||
expect "setting the ${controllerAccountName} service account to an enabled Viewer" "$status" 200
|
||||
echo "set the ${controllerAccountName} service account back to an enabled Viewer"
|
||||
fi
|
||||
|
||||
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
|
||||
echo "could not log in to swarm-bao with Grafana's certificate; leaving swarm-controller's Grafana token as it is." >&2
|
||||
cat "$err" >&2
|
||||
exit 1
|
||||
fi
|
||||
export BAO_TOKEN
|
||||
|
||||
# Checked against the call the controller makes with it.
|
||||
if stored="$(bao kv get -field=value ${lib.escapeShellArg controllerTokenStorePath} 2>"$err")" && [ -n "$stored" ]; then
|
||||
status="$(grafana "Bearer $stored" GET /api/prometheus/grafana/api/v1/alerts)"
|
||||
case "$status" in
|
||||
200) exit 0 ;;
|
||||
401) echo "Grafana refuses the stored token for ${controllerAccountName}; replacing it." ;;
|
||||
*) expect "reading alerts with the stored token" "$status" 200 ;;
|
||||
esac
|
||||
fi
|
||||
|
||||
status="$(grafana "$admin" GET "/api/serviceaccounts/$id/tokens")"
|
||||
expect "listing the ${controllerAccountName} tokens" "$status" 200
|
||||
for token_id in $(jq -r '.[].id' "$body"); do
|
||||
status="$(grafana "$admin" DELETE "/api/serviceaccounts/$id/tokens/$token_id")"
|
||||
expect "deleting token $token_id of ${controllerAccountName}" "$status" 200
|
||||
done
|
||||
|
||||
status="$(grafana "$admin" POST "/api/serviceaccounts/$id/tokens" \
|
||||
--data "{\"name\":\"${controllerAccountName}-$(date +%s)\"}")"
|
||||
expect "minting a ${controllerAccountName} token" "$status" 200
|
||||
key="$(jq -r .key "$body")"
|
||||
: >"$body"
|
||||
if ! printf '%s' "$key" | bao kv put ${lib.escapeShellArg controllerTokenStorePath} value=- >/dev/null 2>"$err"; then
|
||||
echo "could not store the new ${controllerAccountName} token at ${controllerTokenStorePath}; the next run replaces it." >&2
|
||||
cat "$err" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "stored a new ${controllerAccountName} token at ${controllerTokenStorePath}"
|
||||
'';
|
||||
};
|
||||
|
||||
systemd.timers.swarm-bao-grafana-controller-token = storeRefreshTimer "check swarm-controller's Grafana token and replace it if Grafana refuses it";
|
||||
|
||||
containers.${cfg.machine} = {
|
||||
autoStart = true;
|
||||
ephemeral = false;
|
||||
|
|
|
|||
|
|
@ -221,10 +221,10 @@ let
|
|||
&& builtins.elem "bao-client-key.pem:/var/lib/swarm-bao-pki/controller-key.pem" s.swarm-controller.serviceConfig.LoadCredential;
|
||||
}
|
||||
{
|
||||
# The forge token comes from the store, so the unit is handed no file for
|
||||
# it, and neither fixture runs the forge: a controller reaches it by name
|
||||
# wherever it runs.
|
||||
name = "the controller is handed no forge token file, and is pointed at the forge away from it";
|
||||
# The forge and Grafana tokens come from the store, so the unit is handed
|
||||
# no file for either, and neither fixture runs the forge or Grafana: a
|
||||
# controller reaches both by name wherever they run.
|
||||
name = "the controller is handed no forge token file, and is pointed at the forge and Grafana away from it";
|
||||
ok =
|
||||
lib.all
|
||||
(
|
||||
|
|
@ -235,6 +235,7 @@ let
|
|||
!(lib.any (c: lib.hasPrefix "forge-token:" c) u.serviceConfig.LoadCredential)
|
||||
&& !(u.environment ? SWARM_CONTROLLER_FORGE_TOKEN_FILE)
|
||||
&& u.environment.SWARM_CONTROLLER_FORGE_URL == "https://forge.t.local"
|
||||
&& u.environment.SWARM_CONTROLLER_GRAFANA_URL == "https://grafana.t.local"
|
||||
)
|
||||
[
|
||||
baoControllerHere
|
||||
|
|
|
|||
|
|
@ -597,8 +597,10 @@ let
|
|||
#
|
||||
# The admin password and the metrics bearer are the leaves it may create,
|
||||
# and without `update`: a grant that could replace one would let a second
|
||||
# run rotate a value Grafana or the metrics store already holds.
|
||||
name = "the Grafana reader reads its own client secret and creates its admin password and metrics bearer, and nothing else";
|
||||
# run rotate a value Grafana or the metrics store already holds. The
|
||||
# controller's Grafana token is the one leaf it may replace, which it
|
||||
# does when Grafana refuses the stored token.
|
||||
name = "the Grafana reader reads its own client secret, creates its admin password and metrics bearer, and writes the controller's Grafana token, and nothing else";
|
||||
ok =
|
||||
let
|
||||
s = baoGrantHere.systemd.services.swarm-bao-grafana-oidc-policy.script;
|
||||
|
|
@ -622,6 +624,14 @@ let
|
|||
"read"
|
||||
];
|
||||
}
|
||||
{
|
||||
path = "secret/data/swarm/controller/swarm-controller/grafana/token";
|
||||
caps = [
|
||||
"create"
|
||||
"read"
|
||||
"update"
|
||||
];
|
||||
}
|
||||
]
|
||||
&& !(lib.hasInfix "secret/data/swarm/agents" s)
|
||||
&& !(lib.hasInfix "secret/data/swarm/hives" s)
|
||||
|
|
@ -938,18 +948,27 @@ let
|
|||
&& !(noReaders ? swarm-bao-nats-callout-xkey);
|
||||
}
|
||||
{
|
||||
# The admin password is created under Grafana's leaf, so its unit waits
|
||||
# for the unit writing that leaf's role and policy; before it, the create
|
||||
# is refused. The second arm is the absence control.
|
||||
name = "grafana's admin password unit is ordered after grafana's policy unit";
|
||||
# The admin password and the controller's Grafana token are written under
|
||||
# Grafana's leaf, so their units wait for the unit writing that leaf's
|
||||
# role and policy; before it, the write is refused. The last arm is the
|
||||
# absence control.
|
||||
name = "grafana's admin password and controller token units are ordered after grafana's policy unit";
|
||||
ok =
|
||||
let
|
||||
u = baoGrantWithConsumers.systemd.services.swarm-bao-grafana-admin;
|
||||
in
|
||||
lib.elem "swarm-bao-grafana-oidc-policy.service" u.after
|
||||
&& lib.elem "swarm-bao-grafana-oidc-policy.service" u.wants
|
||||
&& !(lib.elem "swarm-bao-grafana-oidc-policy.service" u.requires)
|
||||
&& !(baoGrantNoReaders.systemd.services ? swarm-bao-grafana-admin);
|
||||
lib.all
|
||||
(
|
||||
unit:
|
||||
let
|
||||
u = baoGrantWithConsumers.systemd.services.${unit};
|
||||
in
|
||||
lib.elem "swarm-bao-grafana-oidc-policy.service" u.after
|
||||
&& lib.elem "swarm-bao-grafana-oidc-policy.service" u.wants
|
||||
&& !(lib.elem "swarm-bao-grafana-oidc-policy.service" u.requires)
|
||||
&& !(baoGrantNoReaders.systemd.services ? ${unit})
|
||||
)
|
||||
[
|
||||
"swarm-bao-grafana-admin"
|
||||
"swarm-bao-grafana-controller-token"
|
||||
];
|
||||
}
|
||||
{
|
||||
# The ordering is set apart from each reader's own definition, so it can
|
||||
|
|
@ -1100,6 +1119,7 @@ let
|
|||
"swarm-bao-forwarder-oidc"
|
||||
"swarm-bao-grafana-admin"
|
||||
"swarm-bao-grafana-metrics-bearer"
|
||||
"swarm-bao-grafana-controller-token"
|
||||
]
|
||||
++ policyReaders
|
||||
);
|
||||
|
|
@ -1715,6 +1735,25 @@ let
|
|||
"swarm-bao-forge-publisher-policy"
|
||||
];
|
||||
}
|
||||
{
|
||||
# The controller's Grafana token: read on the one leaf, pinned as the
|
||||
# whole stanza so an added write fails, and named by no policy but the
|
||||
# controller's and Grafana's, which writes it.
|
||||
name = "the controller reads its Grafana token, and only Grafana's grant writes it";
|
||||
ok =
|
||||
let
|
||||
s = baoGrantHere.systemd.services.swarm-bao-controller-policy.script;
|
||||
naming = lib.filter (
|
||||
u: lib.hasInfix "swarm-controller/grafana/token" baoGrantHere.systemd.services.${u}.script
|
||||
) grantingUnitNames;
|
||||
in
|
||||
lib.hasInfix "path \"secret/data/swarm/controller/swarm-controller/grafana/token\" {\n capabilities = [\"read\"]\n}" s
|
||||
&&
|
||||
lib.sort lib.lessThan naming == [
|
||||
"swarm-bao-controller-policy"
|
||||
"swarm-bao-grafana-oidc-policy"
|
||||
];
|
||||
}
|
||||
{
|
||||
# The forge's OIDC client secret by name: the forge publisher reads it,
|
||||
# and the secret publisher writes it under its `swarm/services/*` grant,
|
||||
|
|
|
|||
|
|
@ -249,6 +249,33 @@ let
|
|||
&& lib.hasInfix "admin reset-admin-password --password-from-stdin < /var/lib/grafana-admin/password" service.preStart
|
||||
&& lib.hasInfix "-config ${resetConfig}" service.script;
|
||||
}
|
||||
{
|
||||
# swarm-controller reads alerts with this token, so the account it
|
||||
# belongs to is created and kept a Viewer, the token lands at the path
|
||||
# the controller's grant reads, and an attempt against a Grafana or a
|
||||
# store that never answers ends and retries. The timer re-runs it, so
|
||||
# the unit must not stay active.
|
||||
name = "the controller's Grafana token is a Viewer account's, stored under Grafana's leaf, and each attempt is bounded";
|
||||
ok =
|
||||
let
|
||||
unit = grafanaRemoteAuthelia.systemd.services.swarm-bao-grafana-controller-token;
|
||||
s = unit.script;
|
||||
in
|
||||
!(lib.elem (toString (unit.serviceConfig.TimeoutStartSec or "infinity")) [
|
||||
"infinity"
|
||||
"0"
|
||||
])
|
||||
&& lib.hasInfix "curl --silent --show-error --connect-timeout 5 --max-time 15" s
|
||||
&& lib.hasInfix "--data '{\"name\":\"swarm-controller\",\"role\":\"Viewer\",\"isDisabled\":false}'" s
|
||||
&& lib.hasInfix "--data '{\"role\":\"Viewer\",\"isDisabled\":false}'" s
|
||||
&& lib.hasInfix "bao kv put secret/swarm/controller/swarm-controller/grafana/token value=-" s
|
||||
&& lib.hasInfix "--unix-socket /run/swarm-grafana/grafana.sock" s
|
||||
&& unit.environment.BAO_CLIENT_CERT == "/etc/pki/bao-grafana-oidc.pem"
|
||||
&& unit.serviceConfig.Restart == "on-failure"
|
||||
&& !(unit.serviceConfig.RemainAfterExit or false)
|
||||
&& lib.elem "swarm-bao-grafana-admin.service" unit.after
|
||||
&& grafanaRemoteAuthelia.systemd.timers ? swarm-bao-grafana-controller-token;
|
||||
}
|
||||
{
|
||||
# Same 403-not-a-miss reason as the matrix and queue arms below: the
|
||||
# reader's grant covers the `services` prefix, so a path outside it is
|
||||
|
|
|
|||
|
|
@ -73,6 +73,9 @@ The swarm's control plane. The swarm UI and `swarmctl` are its clients.
|
|||
[`docs/swarm/review-classifier.md`](../docs/swarm/review-classifier.md#swarm-level-settings).
|
||||
- **Relays** — each agent's terminal and turn-state header as SSE, agent
|
||||
icons, the cross-repo issue report, and the UI's quick links.
|
||||
- **Alerts** — the swarm Grafana's current alerts (`GET /api/alerts`), read
|
||||
at request time with a Viewer service-account token from the secret store
|
||||
→ [`docs/swarm/secrets.md`](../docs/swarm/secrets.md).
|
||||
|
||||
It reads its configuration once at startup, from the environment the nix module
|
||||
sets. It persists `webhook-secret` and `operator-matrix-admins.json` in its
|
||||
|
|
|
|||
|
|
@ -508,6 +508,7 @@ pub(crate) mod tests {
|
|||
reviews: Arc::default(),
|
||||
try_outs: crate::review_classifier::TryOuts::default(),
|
||||
runs: crate::review_classifier::runs::Runs::Live(None),
|
||||
alerts: None,
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
391
swarm-controller/src/alerts.rs
Normal file
391
swarm-controller/src/alerts.rs
Normal file
|
|
@ -0,0 +1,391 @@
|
|||
//! `GET /api/alerts`: the alerts the swarm's Grafana holds right now, read at
|
||||
//! request time for swarm-ui's alerts page.
|
||||
//!
|
||||
//! Grafana's URL comes from `swarm-controller.nix`
|
||||
//! (`SWARM_CONTROLLER_GRAFANA_URL`). The token is a Viewer service account's,
|
||||
//! read from the swarm's secret store at
|
||||
//! `swarm_secret_client::grafana::controller_token_path` under this daemon's
|
||||
//! own certificate. `swarm-bao-grafana-controller-token`, on Grafana's host,
|
||||
//! mints it and replaces it whenever Grafana refuses the stored one, so a 401
|
||||
//! here re-reads the store once before giving up.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::future::Future;
|
||||
use std::pin::Pin;
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
|
||||
use anyhow::{Context, Result};
|
||||
use axum::Json;
|
||||
use axum::extract::State;
|
||||
use axum::http::StatusCode;
|
||||
use problem_details::ProblemDetails;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use utoipa::ToSchema;
|
||||
|
||||
use crate::{AppState, error_problem};
|
||||
|
||||
/// The env var `swarm-controller.nix` sets.
|
||||
const URL_ENV: &str = "SWARM_CONTROLLER_GRAFANA_URL";
|
||||
|
||||
/// Grafana's Prometheus-compatible list of alert instances, relative to its
|
||||
/// base URL. Instances in the `Normal` state are not listed.
|
||||
const ALERTS_PATH: &str = "api/prometheus/grafana/api/v1/alerts";
|
||||
|
||||
/// Bound on reaching Grafana.
|
||||
const CONNECT_TIMEOUT: Duration = Duration::from_secs(5);
|
||||
/// Bound on one whole request to Grafana, body included.
|
||||
const TIMEOUT: Duration = Duration::from_secs(15);
|
||||
|
||||
type TokenFuture = Pin<Box<dyn Future<Output = Result<String>> + Send>>;
|
||||
|
||||
/// Where the token comes from: the store in the daemon, a stand-in in tests.
|
||||
pub(crate) type TokenSource = Arc<dyn Fn() -> TokenFuture + Send + Sync>;
|
||||
|
||||
/// One alert instance, as `GET /api/alerts` serves it.
|
||||
#[derive(Clone, Debug, PartialEq, Serialize, ToSchema)]
|
||||
pub(crate) struct Alert {
|
||||
/// The rule's title (Grafana's `alertname` label).
|
||||
pub name: String,
|
||||
/// Grafana's state for the instance: `Alerting`, `Pending`, `NoData` or
|
||||
/// `Error`.
|
||||
pub state: String,
|
||||
/// When the instance entered its state, RFC 3339, as Grafana reports it.
|
||||
pub active_at: Option<String>,
|
||||
/// The rule's `summary` annotation, templated by Grafana.
|
||||
pub summary: Option<String>,
|
||||
/// Every label of the instance, as Grafana reports them.
|
||||
pub labels: BTreeMap<String, String>,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct PromResponse {
|
||||
data: PromData,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct PromData {
|
||||
alerts: Vec<PromAlert>,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct PromAlert {
|
||||
#[serde(default)]
|
||||
labels: BTreeMap<String, String>,
|
||||
#[serde(default)]
|
||||
annotations: BTreeMap<String, String>,
|
||||
state: String,
|
||||
#[serde(rename = "activeAt")]
|
||||
active_at: Option<String>,
|
||||
}
|
||||
|
||||
impl From<PromAlert> for Alert {
|
||||
fn from(a: PromAlert) -> Self {
|
||||
Self {
|
||||
name: a.labels.get("alertname").cloned().unwrap_or_default(),
|
||||
state: a.state,
|
||||
active_at: a.active_at,
|
||||
summary: a.annotations.get("summary").cloned(),
|
||||
labels: a.labels,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Grafana's alerts API, with the token last read from the store.
|
||||
pub(crate) struct Grafana {
|
||||
http: reqwest::Client,
|
||||
alerts_url: url::Url,
|
||||
source: TokenSource,
|
||||
token: tokio::sync::Mutex<Option<String>>,
|
||||
}
|
||||
|
||||
impl Grafana {
|
||||
/// # Errors
|
||||
/// A base URL the alerts path cannot be joined onto, or a client that
|
||||
/// cannot be built.
|
||||
pub(crate) fn new(base: &url::Url, source: TokenSource, timeout: Duration) -> Result<Self> {
|
||||
let http = reqwest::Client::builder()
|
||||
.connect_timeout(CONNECT_TIMEOUT)
|
||||
.timeout(timeout)
|
||||
.build()
|
||||
.context("building the Grafana HTTP client")?;
|
||||
Ok(Self {
|
||||
http,
|
||||
alerts_url: base
|
||||
.join(ALERTS_PATH)
|
||||
.with_context(|| format!("joining {ALERTS_PATH} onto {base}"))?,
|
||||
source,
|
||||
token: tokio::sync::Mutex::new(None),
|
||||
})
|
||||
}
|
||||
|
||||
/// The held token, or a fresh one from the source when `reread` or none is
|
||||
/// held yet.
|
||||
async fn token(&self, reread: bool) -> Result<String> {
|
||||
let mut held = self.token.lock().await;
|
||||
if let Some(t) = held.as_ref().filter(|_| !reread) {
|
||||
return Ok(t.clone());
|
||||
}
|
||||
let fresh = (self.source)().await?;
|
||||
*held = Some(fresh.clone());
|
||||
Ok(fresh)
|
||||
}
|
||||
|
||||
async fn get(&self, token: &str) -> Result<reqwest::Response> {
|
||||
self.http
|
||||
.get(self.alerts_url.clone())
|
||||
.bearer_auth(token)
|
||||
.send()
|
||||
.await
|
||||
.with_context(|| format!("GET {}", self.alerts_url))
|
||||
}
|
||||
|
||||
/// Every alert instance Grafana lists now.
|
||||
///
|
||||
/// # Errors
|
||||
/// No token in the store, Grafana unreachable or slower than the client's
|
||||
/// timeout, a 401 for both the held token and a re-read one, any other
|
||||
/// non-success status, or a body that is not the alerts shape.
|
||||
pub(crate) async fn alerts(&self) -> Result<Vec<Alert>> {
|
||||
let mut res = self.get(&self.token(false).await?).await?;
|
||||
if res.status() == reqwest::StatusCode::UNAUTHORIZED {
|
||||
res = self.get(&self.token(true).await?).await?;
|
||||
}
|
||||
let status = res.status();
|
||||
if !status.is_success() {
|
||||
anyhow::bail!("Grafana answered {status} to GET {}", self.alerts_url);
|
||||
}
|
||||
let body: PromResponse = res
|
||||
.json()
|
||||
.await
|
||||
.with_context(|| format!("reading Grafana's answer to GET {}", self.alerts_url))?;
|
||||
Ok(body.data.alerts.into_iter().map(Alert::from).collect())
|
||||
}
|
||||
}
|
||||
|
||||
/// One login and one read. An empty value is refused like an absent one.
|
||||
async fn token_from_store() -> Result<String> {
|
||||
let path = swarm_secret_client::grafana::controller_token_path()?;
|
||||
let store = crate::store::connect()
|
||||
.await
|
||||
.context("logging in to the swarm secret store")?;
|
||||
let stored: Option<swarm_secret_client::grafana::ControllerToken> = store
|
||||
.read_optional(&path)
|
||||
.await
|
||||
.with_context(|| format!("reading {path}"))?;
|
||||
match stored {
|
||||
Some(t) if !t.value.trim().is_empty() => Ok(t.value.trim().to_owned()),
|
||||
_ => anyhow::bail!(
|
||||
"nothing at {path}: swarm-bao-grafana-controller-token, on the host that runs \
|
||||
Grafana, has not stored this daemon's Grafana token"
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
/// The daemon's Grafana client, or `None` when `SWARM_CONTROLLER_GRAFANA_URL`
|
||||
/// is unset or unusable, which is logged.
|
||||
pub(crate) fn from_env() -> Option<Arc<Grafana>> {
|
||||
let raw = std::env::var(URL_ENV).ok()?;
|
||||
let built = url::Url::parse(&raw)
|
||||
.with_context(|| format!("parse {URL_ENV} ({raw}) as a URL"))
|
||||
.and_then(|base| {
|
||||
let source: TokenSource = Arc::new(|| Box::pin(token_from_store()));
|
||||
Grafana::new(&base, source, TIMEOUT)
|
||||
});
|
||||
match built {
|
||||
Ok(g) => Some(Arc::new(g)),
|
||||
Err(e) => {
|
||||
tracing::warn!(error = %format!("{e:#}"), "Grafana misconfigured; /api/alerts is off");
|
||||
None
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The alerts the swarm's Grafana holds now: firing, pending, and those whose
|
||||
/// rule has no data or failed to evaluate.
|
||||
#[utoipa::path(
|
||||
get,
|
||||
path = "/api/alerts",
|
||||
responses(
|
||||
(status = 200, description = "every alert instance Grafana lists now", body = Vec<Alert>),
|
||||
(status = 503, description = "no Grafana is configured here, its token is not in the store \
|
||||
yet, or Grafana refused, failed or did not answer in time (problem+json)", body = String),
|
||||
),
|
||||
tag = "alerts"
|
||||
)]
|
||||
pub(crate) async fn get_alerts(
|
||||
State(state): State<AppState>,
|
||||
) -> Result<Json<Vec<Alert>>, ProblemDetails> {
|
||||
let Some(grafana) = state.alerts.as_ref() else {
|
||||
return Err(error_problem(
|
||||
StatusCode::SERVICE_UNAVAILABLE,
|
||||
"no Grafana is configured on this host",
|
||||
));
|
||||
};
|
||||
grafana.alerts().await.map(Json).map_err(|e| {
|
||||
let detail = format!("{e:#}");
|
||||
tracing::warn!(error = %detail, "reading Grafana's alerts failed");
|
||||
error_problem(StatusCode::SERVICE_UNAVAILABLE, &detail)
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use axum::body::Body;
|
||||
use axum::http::{HeaderMap, Request};
|
||||
use std::sync::Mutex;
|
||||
use tower::ServiceExt as _;
|
||||
|
||||
/// What the stand-in Grafana answers to a request with the accepted token.
|
||||
const ALERTS_BODY: &str = r#"{"status":"success","data":{"alerts":[{
|
||||
"labels":{"alertname":"swarm queue connect failed","hive":"pr1ma","agent":"atlas"},
|
||||
"annotations":{"summary":"pr1ma/atlas could not connect"},
|
||||
"state":"Alerting","activeAt":"2026-10-15T09:00:00Z","value":"1e+00"}]}}"#;
|
||||
|
||||
/// A stand-in Grafana on a loopback port: the alerts path answers
|
||||
/// [`ALERTS_BODY`] to `Bearer <accepts>` and 401 to anything else, after
|
||||
/// `delay`.
|
||||
async fn fake_grafana(accepts: &'static str, delay: Duration) -> url::Url {
|
||||
let app = axum::Router::new().route(
|
||||
&format!("/{ALERTS_PATH}"),
|
||||
axum::routing::get(move |headers: HeaderMap| async move {
|
||||
tokio::time::sleep(delay).await;
|
||||
let ok = headers
|
||||
.get(axum::http::header::AUTHORIZATION)
|
||||
.is_some_and(|v| v == format!("Bearer {accepts}").as_str());
|
||||
if ok {
|
||||
(StatusCode::OK, ALERTS_BODY)
|
||||
} else {
|
||||
(StatusCode::UNAUTHORIZED, "")
|
||||
}
|
||||
}),
|
||||
);
|
||||
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let addr = listener.local_addr().unwrap();
|
||||
tokio::spawn(async move { axum::serve(listener, app).await });
|
||||
url::Url::parse(&format!("http://{addr}/")).unwrap()
|
||||
}
|
||||
|
||||
/// A token source handing out `tokens` in order, the last one forever,
|
||||
/// counting its calls.
|
||||
fn tokens(tokens: &'static [&'static str]) -> (TokenSource, Arc<Mutex<usize>>) {
|
||||
let calls = Arc::new(Mutex::new(0));
|
||||
let counted = Arc::clone(&calls);
|
||||
let source: TokenSource = Arc::new(move || {
|
||||
let mut n = counted.lock().unwrap();
|
||||
let t = tokens[(*n).min(tokens.len() - 1)];
|
||||
*n += 1;
|
||||
Box::pin(async move { Ok(t.to_owned()) })
|
||||
});
|
||||
(source, calls)
|
||||
}
|
||||
|
||||
async fn get_alerts_with(grafana: Option<Grafana>) -> (StatusCode, serde_json::Value) {
|
||||
let (mut state, _) = crate::tests::state_with_roster();
|
||||
state.alerts = grafana.map(Arc::new);
|
||||
let req = Request::get("/api/alerts")
|
||||
.body(Body::empty())
|
||||
.expect("a request");
|
||||
let res = crate::build_app(state)
|
||||
.oneshot(req)
|
||||
.await
|
||||
.expect("infallible");
|
||||
let status = res.status();
|
||||
let bytes = axum::body::to_bytes(res.into_body(), usize::MAX)
|
||||
.await
|
||||
.expect("a body");
|
||||
(status, serde_json::from_slice(&bytes).unwrap_or_default())
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn serves_grafanas_alerts_read_with_the_stored_token() {
|
||||
let base = fake_grafana("viewer-token", Duration::ZERO).await;
|
||||
let (source, calls) = tokens(&["viewer-token"]);
|
||||
let grafana = Grafana::new(&base, source, TIMEOUT).unwrap();
|
||||
let (status, body) = get_alerts_with(Some(grafana)).await;
|
||||
assert_eq!(status, StatusCode::OK, "{body}");
|
||||
assert_eq!(
|
||||
body,
|
||||
serde_json::json!([{
|
||||
"name": "swarm queue connect failed",
|
||||
"state": "Alerting",
|
||||
"active_at": "2026-10-15T09:00:00Z",
|
||||
"summary": "pr1ma/atlas could not connect",
|
||||
"labels": {
|
||||
"alertname": "swarm queue connect failed",
|
||||
"hive": "pr1ma",
|
||||
"agent": "atlas",
|
||||
},
|
||||
}])
|
||||
);
|
||||
assert_eq!(*calls.lock().unwrap(), 1);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn the_held_token_is_reused_across_requests() {
|
||||
let base = fake_grafana("viewer-token", Duration::ZERO).await;
|
||||
let (source, calls) = tokens(&["viewer-token"]);
|
||||
let grafana = Grafana::new(&base, source, TIMEOUT).unwrap();
|
||||
grafana.alerts().await.unwrap();
|
||||
grafana.alerts().await.unwrap();
|
||||
assert_eq!(*calls.lock().unwrap(), 1);
|
||||
}
|
||||
|
||||
/// The minting unit replaces a token Grafana refuses, so a 401 re-reads
|
||||
/// the store and sends the request once more.
|
||||
#[tokio::test]
|
||||
async fn a_refused_token_is_read_again_from_the_store() {
|
||||
let base = fake_grafana("replaced", Duration::ZERO).await;
|
||||
let (source, calls) = tokens(&["stale", "replaced"]);
|
||||
let grafana = Grafana::new(&base, source, TIMEOUT).unwrap();
|
||||
let alerts = grafana.alerts().await.unwrap();
|
||||
assert_eq!(alerts.len(), 1);
|
||||
assert_eq!(*calls.lock().unwrap(), 2);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_token_refused_twice_is_a_503_naming_the_status() {
|
||||
let base = fake_grafana("replaced", Duration::ZERO).await;
|
||||
let (source, calls) = tokens(&["stale"]);
|
||||
let grafana = Grafana::new(&base, source, TIMEOUT).unwrap();
|
||||
let (status, body) = get_alerts_with(Some(grafana)).await;
|
||||
assert_eq!(status, StatusCode::SERVICE_UNAVAILABLE);
|
||||
assert!(body["detail"].as_str().unwrap().contains("401"), "{body}");
|
||||
assert_eq!(*calls.lock().unwrap(), 2);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_grafana_slower_than_the_timeout_is_a_503() {
|
||||
let base = fake_grafana("viewer-token", Duration::from_secs(5)).await;
|
||||
let (source, _) = tokens(&["viewer-token"]);
|
||||
let grafana = Grafana::new(&base, source, Duration::from_millis(200)).unwrap();
|
||||
let started = std::time::Instant::now();
|
||||
let (status, _) = get_alerts_with(Some(grafana)).await;
|
||||
assert_eq!(status, StatusCode::SERVICE_UNAVAILABLE);
|
||||
assert!(started.elapsed() < Duration::from_secs(4));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn no_token_in_the_store_is_a_503_saying_so() {
|
||||
let base = fake_grafana("viewer-token", Duration::ZERO).await;
|
||||
let source: TokenSource =
|
||||
Arc::new(|| Box::pin(async { anyhow::bail!("nothing at the token path") }));
|
||||
let grafana = Grafana::new(&base, source, TIMEOUT).unwrap();
|
||||
let (status, body) = get_alerts_with(Some(grafana)).await;
|
||||
assert_eq!(status, StatusCode::SERVICE_UNAVAILABLE);
|
||||
assert!(
|
||||
body["detail"]
|
||||
.as_str()
|
||||
.unwrap()
|
||||
.contains("nothing at the token path"),
|
||||
"{body}"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn no_grafana_configured_is_a_503() {
|
||||
let (status, _) = get_alerts_with(None).await;
|
||||
assert_eq!(status, StatusCode::SERVICE_UNAVAILABLE);
|
||||
}
|
||||
}
|
||||
|
|
@ -168,6 +168,7 @@ mod tests {
|
|||
try_outs: super::super::review_classifier::TryOuts::default(),
|
||||
reviews: std::sync::Arc::default(),
|
||||
runs: crate::review_classifier::runs::Runs::Live(None),
|
||||
alerts: None,
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -138,6 +138,7 @@ mod tests {
|
|||
try_outs: super::super::review_classifier::TryOuts::default(),
|
||||
reviews: std::sync::Arc::default(),
|
||||
runs: crate::review_classifier::runs::Runs::Live(None),
|
||||
alerts: None,
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -973,6 +973,7 @@ pub(crate) mod tests {
|
|||
try_outs: super::super::review_classifier::TryOuts::default(),
|
||||
reviews: std::sync::Arc::default(),
|
||||
runs: crate::review_classifier::runs::Runs::Live(None),
|
||||
alerts: None,
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -50,6 +50,7 @@ mod agent_login;
|
|||
mod agent_renewal;
|
||||
mod agent_state_stream;
|
||||
mod agent_status;
|
||||
mod alerts;
|
||||
mod auth;
|
||||
mod builder;
|
||||
mod claude_login;
|
||||
|
|
@ -839,6 +840,7 @@ fn socket_path() -> PathBuf {
|
|||
(name = "schedules", description = "scheduled prompts (`swarmctl schedule`)"),
|
||||
(name = "review-classifier", description = "trying review classifications against a PR (swarm-ui's try-out page), and the classifier's swarm-wide settings"),
|
||||
(name = "pulls", description = "open pull requests, for the try-out page's dropdowns"),
|
||||
(name = "alerts", description = "the swarm Grafana's current alerts (swarm-ui's alerts page)"),
|
||||
)
|
||||
)]
|
||||
struct ApiDoc;
|
||||
|
|
@ -972,6 +974,9 @@ struct AppState {
|
|||
reviews: Arc<review_classifier::review::Reviews>,
|
||||
/// The review classifier's finished runs.
|
||||
runs: review_classifier::runs::Runs,
|
||||
/// Grafana's alerts API, for `GET /api/alerts`. `None` when no Grafana URL
|
||||
/// is configured.
|
||||
alerts: Option<Arc<alerts::Grafana>>,
|
||||
}
|
||||
|
||||
/// Env var the controller's NixOS module sets from
|
||||
|
|
@ -3451,6 +3456,7 @@ async fn main() -> Result<()> {
|
|||
try_outs: review_classifier::TryOuts::default(),
|
||||
reviews: Arc::default(),
|
||||
runs,
|
||||
alerts: alerts::from_env(),
|
||||
};
|
||||
|
||||
let app = build_app(state);
|
||||
|
|
@ -3468,6 +3474,7 @@ async fn main() -> Result<()> {
|
|||
fn build_app(state: AppState) -> axum::Router {
|
||||
let (router, api) = OpenApiRouter::<AppState>::with_openapi(ApiDoc::openapi())
|
||||
.routes(routes!(health))
|
||||
.routes(routes!(alerts::get_alerts))
|
||||
.routes(routes!(get_hives))
|
||||
.routes(routes!(get_hives_status))
|
||||
.routes(routes!(get_links))
|
||||
|
|
@ -3660,6 +3667,7 @@ mod tests {
|
|||
try_outs: super::review_classifier::TryOuts::default(),
|
||||
reviews: std::sync::Arc::default(),
|
||||
runs: super::review_classifier::runs::Runs::Live(None),
|
||||
alerts: None,
|
||||
};
|
||||
(state, sched)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -585,6 +585,7 @@ mod tests {
|
|||
try_outs: super::super::review_classifier::TryOuts::default(),
|
||||
reviews: std::sync::Arc::default(),
|
||||
runs: crate::review_classifier::runs::Runs::Live(None),
|
||||
alerts: None,
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
54
swarm-secret-client/src/grafana.rs
Normal file
54
swarm-secret-client/src/grafana.rs
Normal file
|
|
@ -0,0 +1,54 @@
|
|||
//! The Grafana agreement: where `swarm-controller`'s token for Grafana's API
|
||||
//! lives, and what the object at that path holds.
|
||||
//!
|
||||
//! `swarm-bao-grafana-controller-token`, on the host that runs Grafana, mints
|
||||
//! the token for a Viewer service account and writes it here under Grafana's
|
||||
//! own certificate; `swarm-controller` reads it under its own. Those two
|
||||
//! principals' stanzas are the only ones in the store that cover this path.
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use crate::{
|
||||
Error,
|
||||
path::{Kind, principal_prefix},
|
||||
};
|
||||
|
||||
/// The path holding `swarm-controller`'s Grafana token.
|
||||
///
|
||||
/// Under [`Kind::Controller`], which no hive's policy reads. The nix half is
|
||||
/// `swarm-bao.nix`'s `controllerGrafanaTokenLeaf`.
|
||||
///
|
||||
/// # Errors
|
||||
/// Never in practice: the name segment is a constant. The `Result` is
|
||||
/// [`principal_prefix`]'s.
|
||||
pub fn controller_token_path() -> Result<String, Error> {
|
||||
let prefix = principal_prefix(Kind::Controller, crate::matrix::CONTROLLER)?;
|
||||
Ok(format!("{prefix}/grafana/token"))
|
||||
}
|
||||
|
||||
/// What [`controller_token_path`] holds.
|
||||
///
|
||||
/// No `Debug` derive: `value` reads every alert and dashboard in the swarm's
|
||||
/// Grafana, and a derived `Debug` is one `{:?}` away from a log line.
|
||||
#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct ControllerToken {
|
||||
/// The token itself. The minting unit writes it with
|
||||
/// `bao kv put … value=-`, so this name is load-bearing for a writer this
|
||||
/// crate does not control.
|
||||
pub value: String,
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn the_controllers_grafana_token_is_under_the_controllers_prefix() {
|
||||
// The literal is what `swarm-bao.nix` grants Grafana's role and the
|
||||
// controller's, and what the minting unit writes.
|
||||
assert_eq!(
|
||||
controller_token_path().expect("a constant segment is legal"),
|
||||
"swarm/controller/swarm-controller/grafana/token"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
@ -5,10 +5,10 @@
|
|||
//! the rules every path obeys ([`path`]), the translation from this
|
||||
//! deployment's environment into a logged-in client ([`client`]), and, per kind
|
||||
//! of secret, the path it lives at together with the fields it holds
|
||||
//! ([`matrix`], [`queue`], [`mtls`], [`forge`], [`github`], [`acp`], [`ci`],
|
||||
//! [`review_classifier`], [`builder`], [`claude_login`]). Each of those is a
|
||||
//! thing the controller and a hive must say identically, so it is said once
|
||||
//! here.
|
||||
//! ([`matrix`], [`queue`], [`mtls`], [`forge`], [`github`], [`grafana`],
|
||||
//! [`acp`], [`ci`], [`review_classifier`], [`builder`], [`claude_login`]).
|
||||
//! Each of those is a thing the controller and a hive must say identically, so
|
||||
//! it is said once here.
|
||||
//!
|
||||
//! [`policy`] is the same kind of agreement seen from the other side: which of
|
||||
//! those paths a given principal's own token may read — a hive's, and an
|
||||
|
|
@ -31,6 +31,7 @@ pub mod claude_login;
|
|||
pub mod client;
|
||||
pub mod forge;
|
||||
pub mod github;
|
||||
pub mod grafana;
|
||||
pub mod matrix;
|
||||
pub mod mtls;
|
||||
pub mod path;
|
||||
|
|
|
|||
Loading…
Reference in a new issue