Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
3b4f5d60ff | ||
|
|
7d09776742 |
25 changed files with 891 additions and 38 deletions
|
|
@ -63,6 +63,7 @@ of the cell says how.
|
|||
| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass |
|
||||
| `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, at start and on refusal | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read at start and held in memory; read again when authelia refuses it, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value |
|
||||
| `swarm/controller/swarm-controller/forge/token` | the forge, inside the `hive-forge` container, once; `swarm-bao-forge-publisher` on the forge's host copies it in under its own certificate, with a write-only grant | `swarm-controller`, under its own certificate, at start and on refusal | ❌ the forge mints it once and keeps its copy in the container; the publisher writes that copy each time the forge container starts and each time the mint writes the file | ✅ the controller reads it at start, and in the background with backoff until it first finds it. When the forge answers 401 it reads it again and sends the request once more with the new token, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value |
|
||||
| `swarm/controller/swarm-controller/grafana/token` | `swarm-bao-grafana-controller-token` on Grafana's host, under Grafana's certificate: a token for the `swarm-controller` Viewer service account, minted with Grafana's admin password | `swarm-controller`, under its own certificate, on the first `/api/alerts` request and again when Grafana refuses it | ✅ a timer starts the unit again 10 minutes after each run; it keeps a stored token Grafana accepts, and mints a new one when Grafana answers 401 or the store holds none | ✅ the controller holds the token in memory; when Grafana answers 401 it reads the store once more and retries the request |
|
||||
| `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config; then `hive-agent-bao-identity` in the agent container, under the leaf it holds | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ✅ `hive-agent-bao-identity` re-reads it every 10 minutes and swaps a new leaf into `/run/hive-agent-bao-identity/`, which every store login in the container presents. It reads under the leaf it holds, so it renews only while that leaf is valid |
|
||||
| `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive; `swarm-logs` in the agent container, under the same certificate, on every query, as the bearer it presents at the log store; `swarm-nats-auth` and `swarm-logs-auth`, each under its own certificate, to check a presented secret | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff. `swarm-logs` reads it on every query and once more after a 401. `swarm-logs-auth` keeps granting a secret it verified for `deploy.victorialogs.authVerdictTtl` (60 s by default) without reading the path, so a replaced or deleted secret still works at the log store for up to that long |
|
||||
| `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer |
|
||||
|
|
|
|||
|
|
@ -60,6 +60,7 @@ neither is a renaming of the other.
|
|||
| a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) |
|
||||
| the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf** | `/var/lib/hive-otel-oidc/hive-<name>.secret` on the hive's host, 0400 root, loaded by `LoadCredential` | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) |
|
||||
| the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | `/var/lib/forgejo/swarm-controller-token` inside the forge container, the minter's record that it has minted; `swarm-bao-forge-publisher` copies it into the swarm secret store at `swarm/controller/swarm-controller/forge/token` each time the forge container starts and each time the mint writes the file; the controller holds it in memory only | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) |
|
||||
| the swarm controller's Grafana token | `swarm-bao-grafana-controller-token.service`, **on the host that runs Grafana**, as Grafana's built-in admin over its socket: it creates the `swarm-controller` service account as a Viewer (and sets it back to Viewer on every run), mints a token for it and writes it to the swarm secret store at `swarm/controller/swarm-controller/grafana/token` under Grafana's leaf. Re-run every ten minutes: it keeps a stored token Grafana accepts, and replaces one Grafana refuses with 401 (or a missing one), deleting every other token of that account first | the swarm secret store only; the controller reads it for `GET /api/alerts` and holds it in memory, re-reading the store once when Grafana answers 401 | nothing to place: Grafana's leaf is already required to run Grafana. Until the unit has run, `/api/alerts` answers 503 naming the empty path |
|
||||
| an agent's claude login (`.credentials.json`) | `swarm-claude-login.service`, in its container, when an operator completes `claude auth login` from the agent's card in the swarm UI; it stores the file as a new version at `swarm/agents/<agent>/claude-credentials` under its own leaf, which may create and update that path and read nothing under an agent | the swarm secret store at that path; the agent's `hive-agent` reads it under the agent's own leaf every 30 seconds while it waits for a login and writes it to `~/.claude/.credentials.json` | the same read, co-located or not; nothing for an operator to place beyond the agent's store leaf. When the store's copy and the agent's differ the KV version decides: the agent installs only a version newer than the last it installed (`claude-credentials-version` under the harness dir), because claude rotates the refresh token in its own copy and the stored one goes stale. The agent ignores a stale or equal version, and a relogin writes a newer one — [claude-login.md](claude-login.md) |
|
||||
| the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
|
||||
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
|
||||
|
|
|
|||
|
|
@ -7,17 +7,18 @@ domain, covers host-level detail for one hive.
|
|||
|
||||
## What it shows
|
||||
|
||||
| route | what |
|
||||
| ---------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `/` | the start page: the swarm's name over a digital-rain backdrop; the swarm name in the header links here |
|
||||
| `/hives` | the hive directory, each hive with its last reported status |
|
||||
| `/agents` | every agent: status, config PR, wanted state, inference settings, resource caps; create agents, link forge, matrix and GitHub accounts |
|
||||
| `/agents/<name>/terminal` | one agent's live terminal |
|
||||
| `/agents/<name>/subagents/<subagent>/terminal` | one of that agent's subagents' live terminal |
|
||||
| `/jobs` | the controller's job graph — where agent creation and credential mints show progress |
|
||||
| `/schedules` | the scheduled prompts: add, edit, pause, resume and remove |
|
||||
| `/issues` | a cross-repo issue report |
|
||||
| `/review` | the review classifier: its recent runs, a try-out on a real PR, and its settings |
|
||||
| route | what |
|
||||
| ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `/` | the start page: the swarm's name over a digital-rain backdrop; the swarm name in the header links here |
|
||||
| `/hives` | the hive directory, each hive with its last reported status |
|
||||
| `/agents` | every agent: status, config PR, wanted state, inference settings, resource caps; create agents, link forge, matrix and GitHub accounts |
|
||||
| `/agents/<name>/terminal` | one agent's live terminal |
|
||||
| `/agents/<name>/subagents/<subagent>/terminal` | one of that agent's subagents' live terminal |
|
||||
| `/alerts` | the alerts the swarm's Grafana holds now (firing, pending, no data, error), read through the controller with a Viewer service-account token |
|
||||
| `/jobs` | the controller's job graph — where agent creation and credential mints show progress |
|
||||
| `/schedules` | the scheduled prompts: add, edit, pause, resume and remove |
|
||||
| `/issues` | a cross-repo issue report |
|
||||
| `/review` | the review classifier: its recent runs, a try-out on a real PR, and its settings |
|
||||
|
||||
Everything it shows comes from [`swarm-controller`](../../swarm-controller/README.md).
|
||||
An agent created here or with `swarmctl agent create` starts `paused`; set it
|
||||
|
|
|
|||
|
|
@ -4,6 +4,7 @@ import { Route, Switch } from "wouter-preact";
|
|||
import { Shell } from "./shell/Shell.js";
|
||||
import { AgentsPage } from "./pages/agents/AgentsPage.js";
|
||||
import { AgentTerminalPage } from "./pages/agents/AgentTerminalPage.js";
|
||||
import { AlertsPage } from "./pages/AlertsPage.js";
|
||||
import { ComponentsPage } from "./pages/ComponentsPage.js";
|
||||
import { JobsPage } from "./pages/JobsPage.js";
|
||||
import { HivesPage } from "./pages/HivesPage.js";
|
||||
|
|
@ -27,6 +28,7 @@ export function App() {
|
|||
<Switch>
|
||||
<Route path="/" component={StartPage} />
|
||||
<Route path="/hives" component={HivesPage} />
|
||||
<Route path="/alerts" component={AlertsPage} />
|
||||
<Route path="/agents" component={AgentsPage} />
|
||||
<Route path="/agents/:name/terminal" component={AgentTerminalPage} />
|
||||
<Route
|
||||
|
|
|
|||
138
frontend/packages/swarm-ui/src/pages/AlertsPage.tsx
Normal file
138
frontend/packages/swarm-ui/src/pages/AlertsPage.tsx
Normal file
|
|
@ -0,0 +1,138 @@
|
|||
// <AlertsPage> — `/alerts`' content: the alerts the swarm's Grafana holds
|
||||
// now, from swarm-controller's `GET /api/alerts`. Same shape as `HivesPage`:
|
||||
// one fetch on a `RefreshIntervalPicker` cadence, rendered as a `Table`.
|
||||
import { useState } from "preact/hooks";
|
||||
import { ApiErrorPanel } from "@hive/shared/api-error-panel.js";
|
||||
import { readApiError, type ProblemDetails } from "@hive/shared/api-error.js";
|
||||
import { Badge, type BadgeTone } from "@hive/shared/badge.js";
|
||||
import { Panel } from "../ui/panel/Panel.js";
|
||||
import { RelativeTime } from "../ui/relative-time/RelativeTime.js";
|
||||
import {
|
||||
RefreshIntervalPicker,
|
||||
useRefreshInterval,
|
||||
type RefreshIntervalMs,
|
||||
} from "../ui/refresh-interval/RefreshInterval.js";
|
||||
import { Table, type TableColumn } from "../ui/table/Table.js";
|
||||
|
||||
interface Alert {
|
||||
name: string;
|
||||
// Grafana's state for the instance: `Alerting`, `Pending`, `NoData` or
|
||||
// `Error`. Anything else renders neutral.
|
||||
state: string;
|
||||
active_at: string | null;
|
||||
summary: string | null;
|
||||
labels: Record<string, string>;
|
||||
}
|
||||
|
||||
const STATE_TONE: Record<string, BadgeTone> = {
|
||||
Alerting: "negative",
|
||||
Error: "negative",
|
||||
NoData: "warning",
|
||||
Pending: "warning",
|
||||
};
|
||||
|
||||
// Grafana reports an instance that never became active with the zero time,
|
||||
// which `Date.parse` reads as year 1.
|
||||
function activeSinceMs(a: Alert): number | null {
|
||||
if (a.active_at === null) return null;
|
||||
const ms = Date.parse(a.active_at);
|
||||
return Number.isNaN(ms) || ms <= 0 ? null : ms;
|
||||
}
|
||||
|
||||
// A row needs a key that is unique among instances of the same rule.
|
||||
function alertKey(a: Alert): string {
|
||||
return JSON.stringify(Object.entries(a.labels).sort());
|
||||
}
|
||||
|
||||
const COLUMNS: TableColumn<Alert>[] = [
|
||||
{
|
||||
key: "name",
|
||||
header: "alert",
|
||||
render: (a) => a.name,
|
||||
sortBy: (a) => a.name,
|
||||
filterValue: (a) => a.name,
|
||||
},
|
||||
{
|
||||
key: "state",
|
||||
header: "state",
|
||||
render: (a) => {
|
||||
const since = activeSinceMs(a);
|
||||
return (
|
||||
<Badge
|
||||
tone={STATE_TONE[a.state] ?? "neutral"}
|
||||
value={
|
||||
since !== null ? (
|
||||
<>
|
||||
{a.state} (<RelativeTime epochMs={since} />)
|
||||
</>
|
||||
) : (
|
||||
a.state
|
||||
)
|
||||
}
|
||||
/>
|
||||
);
|
||||
},
|
||||
sortBy: (a) => a.state,
|
||||
filterValues: (a) => [a.state],
|
||||
filterMode: "multiselect",
|
||||
},
|
||||
{
|
||||
key: "summary",
|
||||
header: "summary",
|
||||
render: (a) => a.summary ?? "—",
|
||||
sortBy: (a) => a.summary ?? "",
|
||||
filterValue: (a) => a.summary ?? "",
|
||||
},
|
||||
];
|
||||
|
||||
const DEFAULT_INTERVAL_MS: RefreshIntervalMs = 30_000;
|
||||
|
||||
export function AlertsPage() {
|
||||
const [alerts, setAlerts] = useState<Alert[] | null>(null);
|
||||
const [error, setError] = useState<ProblemDetails | null>(null);
|
||||
const [intervalMs, setIntervalMs] =
|
||||
useState<RefreshIntervalMs>(DEFAULT_INTERVAL_MS);
|
||||
|
||||
useRefreshInterval(intervalMs, () => {
|
||||
(async () => {
|
||||
const r = await fetch("/api/alerts");
|
||||
if (!r.ok) {
|
||||
setError(await readApiError(r));
|
||||
return;
|
||||
}
|
||||
setAlerts((await r.json()) as Alert[]);
|
||||
setError(null);
|
||||
})().catch((e: unknown) => setError({ detail: String(e) }));
|
||||
});
|
||||
|
||||
return (
|
||||
<Panel
|
||||
title="alerts"
|
||||
icon="🚨"
|
||||
actions={
|
||||
<RefreshIntervalPicker
|
||||
id="alerts-refresh"
|
||||
value={intervalMs}
|
||||
onChange={setIntervalMs}
|
||||
/>
|
||||
}
|
||||
>
|
||||
{error ? (
|
||||
<ApiErrorPanel
|
||||
context="failed to load Grafana's alerts"
|
||||
problem={error}
|
||||
/>
|
||||
) : null}
|
||||
{!error && alerts === null ? <p>loading…</p> : null}
|
||||
{alerts && alerts.length === 0 ? <p>no alerts.</p> : null}
|
||||
{alerts && alerts.length > 0 ? (
|
||||
<Table
|
||||
columns={COLUMNS}
|
||||
rows={alerts}
|
||||
rowKey={alertKey}
|
||||
storageKey="swarm-ui:alerts:table-filters"
|
||||
/>
|
||||
) : null}
|
||||
</Panel>
|
||||
);
|
||||
}
|
||||
|
|
@ -51,6 +51,7 @@ const MOTION_KEY = "swarm-ui:motion-override";
|
|||
const NAV_ITEMS: { href: string; label: string; accent: string }[] = [
|
||||
{ href: "/hives", label: "hives", accent: "var(--purple)" },
|
||||
{ href: "/agents", label: "agents", accent: "var(--green)" },
|
||||
{ href: "/alerts", label: "alerts", accent: "var(--red)" },
|
||||
{ href: "/jobs", label: "jobs", accent: "var(--pink)" },
|
||||
{ href: "/schedules", label: "schedules", accent: "var(--cyan)" },
|
||||
{ href: "/issues", label: "issues", accent: "var(--yellow)" },
|
||||
|
|
|
|||
|
|
@ -67,7 +67,8 @@ let
|
|||
];
|
||||
};
|
||||
# ./swarm-grafana.nix, under Grafana's leaf, which may create the admin
|
||||
# password and the metrics bearer only once that policy is written.
|
||||
# password, the metrics bearer and the controller's Grafana token only
|
||||
# once that policy is written.
|
||||
swarm-bao-grafana-admin = {
|
||||
here = deployCfg.grafana.enable;
|
||||
policies = [ "swarm-bao-grafana-oidc-policy" ];
|
||||
|
|
@ -76,6 +77,10 @@ let
|
|||
here = deployCfg.grafana.enable;
|
||||
policies = [ "swarm-bao-grafana-oidc-policy" ];
|
||||
};
|
||||
swarm-bao-grafana-controller-token = {
|
||||
here = deployCfg.grafana.enable;
|
||||
policies = [ "swarm-bao-grafana-oidc-policy" ];
|
||||
};
|
||||
# ./hive-matrix.nix, under matrix-ctl's leaf.
|
||||
swarm-bao-matrix-oidc = {
|
||||
here = deployCfg.matrix.enable;
|
||||
|
|
|
|||
|
|
@ -47,6 +47,7 @@ let
|
|||
swarm-bao-grafana-oidc = deployCfg.grafana.enable;
|
||||
swarm-bao-grafana-admin = deployCfg.grafana.enable;
|
||||
swarm-bao-grafana-metrics-bearer = deployCfg.grafana.enable;
|
||||
swarm-bao-grafana-controller-token = deployCfg.grafana.enable;
|
||||
# ./swarm-nats.nix
|
||||
swarm-bao-nats-callout-xkey = deployCfg.nats.enable;
|
||||
# ./swarm-ci-runner.nix
|
||||
|
|
|
|||
|
|
@ -369,9 +369,9 @@ let
|
|||
# Each hive's matrix sender token (`matrix_account::hive_sender`) grants `read`
|
||||
# for the same keep-if-live reason, and `+` for the same glob-only-as-last-segment reason.
|
||||
#
|
||||
# Three leaves it reads and never writes: the swarm appservice token
|
||||
# (matrix-ctl, `matrixCtlPolicyText` below), its own OIDC client secret (the
|
||||
# publisher) and its own forge token (the forge's host, `forgePublisherWriters`).
|
||||
# Four leaves it reads and never writes: the swarm appservice token (matrix-ctl),
|
||||
# its OIDC client secret (the publisher), its forge token (the forge's host)
|
||||
# and its Grafana token (Grafana's host). Each writer's grant is named below.
|
||||
#
|
||||
# The agent PKI grant is its only path on that mount: it can ask the one role
|
||||
# for a certificate, not write that role or reach the issuer.
|
||||
|
|
@ -457,6 +457,10 @@ let
|
|||
capabilities = ["read"]
|
||||
}
|
||||
|
||||
path "${credentialMountPath}/data/${controllerGrafanaTokenLeaf}" {
|
||||
capabilities = ["read"]
|
||||
}
|
||||
|
||||
path "${agentPkiMountPath}/issue/${agentPkiRoleName}" {
|
||||
capabilities = ["update"]
|
||||
}
|
||||
|
|
@ -582,6 +586,11 @@ let
|
|||
# it; the controller reads it and holds no other copy.
|
||||
controllerForgeTokenLeaf = "swarm/controller/swarm-controller/forge/token";
|
||||
|
||||
# The controller's Viewer token for Grafana's API, the nix half of
|
||||
# `swarm_secret_client::grafana::controller_token_path`. Grafana's host
|
||||
# writes it; the controller reads it and holds no other copy.
|
||||
controllerGrafanaTokenLeaf = "swarm/controller/swarm-controller/grafana/token";
|
||||
|
||||
# The forge's runner registration token, the nix half of
|
||||
# `swarm_secret_client::ci::registration_token_path`. The controller writes
|
||||
# it and cannot read it back (`forge::runner_token` keeps the value it last
|
||||
|
|
@ -911,6 +920,10 @@ let
|
|||
# in ./swarm-grafana.nix) and the bearer its metrics datasource presents
|
||||
# (`swarm-bao-grafana-metrics-bearer`). `create` without `update`, so a stored
|
||||
# value is never replaced.
|
||||
#
|
||||
# The controller's Grafana token is the one it may replace:
|
||||
# `swarm-bao-grafana-controller-token` mints a new one when Grafana refuses
|
||||
# the stored one, as it does after its database is recreated.
|
||||
grafanaOidcReaders = [
|
||||
{
|
||||
name = "swarm-grafana-oidc";
|
||||
|
|
@ -918,7 +931,12 @@ let
|
|||
policyText =
|
||||
readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.grafana.oidc.clientId}/oidc/client"
|
||||
+ createStanza "${credentialMountPath}/data/${grafanaAdminLeaf}"
|
||||
+ createStanza "${credentialMountPath}/data/${grafanaMetricsBearerLeaf}";
|
||||
+ createStanza "${credentialMountPath}/data/${grafanaMetricsBearerLeaf}"
|
||||
+ ''
|
||||
path "${credentialMountPath}/data/${controllerGrafanaTokenLeaf}" {
|
||||
capabilities = ["create", "read", "update"]
|
||||
}
|
||||
'';
|
||||
}
|
||||
];
|
||||
|
||||
|
|
|
|||
|
|
@ -811,6 +811,11 @@ in
|
|||
# separated: a matrix user id cannot contain a space. Consumed by
|
||||
# swarm-controller/src/matrix_account/operator_admin.rs.
|
||||
SWARM_CONTROLLER_MATRIX_ADMIN_SYNC_IGNORE_USERS = lib.concatStringsSep " " matrixCfg.adminSync.ignoreUsers;
|
||||
# Where `GET /api/alerts` reads alerts (swarm-controller/src/alerts.rs).
|
||||
# Every swarm has one Grafana, reached by name wherever it runs. The
|
||||
# token is in the store, written by `swarm-bao-grafana-controller-token`
|
||||
# on Grafana's host.
|
||||
SWARM_CONTROLLER_GRAFANA_URL = "https://${swarmCfg.grafana.domain}";
|
||||
}
|
||||
// queueEnv
|
||||
// forgeEnv
|
||||
|
|
|
|||
|
|
@ -273,6 +273,14 @@ let
|
|||
metricsBearerPath = "/var/lib/grafana-metrics/bearer";
|
||||
hostMetricsBearerPath = "/var/lib/nixos-containers/${cfg.machine}${metricsBearerPath}";
|
||||
|
||||
# swarm-controller's token for Grafana's API, minted by
|
||||
# `swarm-bao-grafana-controller-token` below for a Viewer service account of
|
||||
# this name. `controllerGrafanaTokenLeaf` in ./swarm-bao.nix is the nix half
|
||||
# of the path, `swarm_secret_client::grafana::controller_token_path` the Rust
|
||||
# half.
|
||||
controllerAccountName = "swarm-controller";
|
||||
controllerTokenStorePath = "secret/swarm/controller/swarm-controller/grafana/token";
|
||||
|
||||
socketPath = "${deployCfg.grafana.socketDir}/grafana.sock";
|
||||
|
||||
# Both static NixOS ids, and both checked rather than assumed: nginx has
|
||||
|
|
@ -849,6 +857,148 @@ in
|
|||
'';
|
||||
};
|
||||
|
||||
# swarm-controller reads Grafana's alerts with a token of a Viewer service
|
||||
# account, created here through Grafana's API as the built-in admin and
|
||||
# handed over through the store.
|
||||
#
|
||||
# Re-run by the timer below. A stored token Grafana still accepts is kept;
|
||||
# one it refuses with 401, or none at all, is replaced: every token of the
|
||||
# account is deleted and one new one is minted and stored. The account's
|
||||
# role is set back to Viewer on every run.
|
||||
systemd.services.swarm-bao-grafana-controller-token = {
|
||||
description = "give swarm-controller a Viewer token for Grafana's API through the swarm secret store";
|
||||
# Ordered after the store where it runs here:
|
||||
# ./glue-bao-readers-store-order.nix.
|
||||
after = [
|
||||
"container@${cfg.machine}.service"
|
||||
"swarm-bao-grafana-admin.service"
|
||||
];
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
path = [
|
||||
baoDeploy.package
|
||||
pkgs.coreutils
|
||||
pkgs.curl
|
||||
pkgs.jq
|
||||
];
|
||||
# ./lib/store-retry.nix. A Grafana still starting refuses the
|
||||
# connection, and that attempt fails into the same retry.
|
||||
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
|
||||
serviceConfig = storeRetry.serviceConfig // {
|
||||
# No `RemainAfterExit`: the timer below starts this again.
|
||||
Type = "oneshot";
|
||||
SyslogIdentifier = "swarm-bao-grafana-controller-token";
|
||||
# Grafana's answers, the new token among them, pass through a temp file.
|
||||
PrivateTmp = true;
|
||||
# Each call below is bounded by curl's own timeouts; a run makes up to
|
||||
# about ten of them.
|
||||
TimeoutStartSec = "3min";
|
||||
};
|
||||
environment = {
|
||||
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
|
||||
BAO_CLIENT_CERT = baoDeploy.grafanaOidcClientCertFile;
|
||||
BAO_CLIENT_KEY = baoDeploy.grafanaOidcClientKeyFile;
|
||||
}
|
||||
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
|
||||
BAO_CACERT = baoDeploy.serverCaFile;
|
||||
};
|
||||
script = ''
|
||||
set -euo pipefail
|
||||
|
||||
err="$(mktemp)"
|
||||
body="$(mktemp)"
|
||||
trap 'rm -f "$err" "$body"' EXIT
|
||||
|
||||
# One call to Grafana's API through its socket. Prints the HTTP
|
||||
# status; the body lands in "$body". The credential reaches curl as
|
||||
# a header file, never as an argument.
|
||||
grafana() {
|
||||
local auth="$1" method="$2" path="$3"
|
||||
shift 3
|
||||
curl --silent --show-error --connect-timeout 5 --max-time 15 \
|
||||
--unix-socket ${lib.escapeShellArg socketPath} \
|
||||
-H @<(printf 'Authorization: %s\n' "$auth") \
|
||||
-H 'Content-Type: application/json' \
|
||||
-X "$method" -o "$body" -w '%{http_code}' "$@" \
|
||||
"http://localhost$path"
|
||||
}
|
||||
|
||||
# Fails the attempt unless the last call answered one of the statuses
|
||||
# given.
|
||||
expect() {
|
||||
local what="$1" status="$2"
|
||||
shift 2
|
||||
for ok in "$@"; do
|
||||
[ "$status" = "$ok" ] && return 0
|
||||
done
|
||||
echo "Grafana answered $status to $what:" >&2
|
||||
cat "$body" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
if [ ! -s ${lib.escapeShellArg hostAdminPasswordPath} ]; then
|
||||
echo "no Grafana admin password at ${hostAdminPasswordPath} yet; swarm-bao-grafana-admin writes it." >&2
|
||||
exit 1
|
||||
fi
|
||||
admin="Basic $(printf 'admin:%s' "$(< ${lib.escapeShellArg hostAdminPasswordPath})" | base64 -w0)"
|
||||
|
||||
status="$(grafana "$admin" GET '/api/serviceaccounts/search?perpage=1000&query=${controllerAccountName}')"
|
||||
expect "the service account search" "$status" 200
|
||||
account="$(jq -c --arg n ${controllerAccountName} '[.serviceAccounts[] | select(.name == $n)][0] // empty' "$body")"
|
||||
if [ -z "$account" ]; then
|
||||
status="$(grafana "$admin" POST /api/serviceaccounts \
|
||||
--data '{"name":"${controllerAccountName}","role":"Viewer","isDisabled":false}')"
|
||||
expect "creating the ${controllerAccountName} service account" "$status" 200 201
|
||||
account="$(jq -c . "$body")"
|
||||
echo "created the ${controllerAccountName} service account as a Viewer"
|
||||
fi
|
||||
id="$(jq -r .id <<<"$account")"
|
||||
if [ "$(jq -r '"\(.role) \(.isDisabled // false)"' <<<"$account")" != "Viewer false" ]; then
|
||||
status="$(grafana "$admin" PATCH "/api/serviceaccounts/$id" \
|
||||
--data '{"role":"Viewer","isDisabled":false}')"
|
||||
expect "setting the ${controllerAccountName} service account to an enabled Viewer" "$status" 200
|
||||
echo "set the ${controllerAccountName} service account back to an enabled Viewer"
|
||||
fi
|
||||
|
||||
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
|
||||
echo "could not log in to swarm-bao with Grafana's certificate; leaving swarm-controller's Grafana token as it is." >&2
|
||||
cat "$err" >&2
|
||||
exit 1
|
||||
fi
|
||||
export BAO_TOKEN
|
||||
|
||||
# Checked against the call the controller makes with it.
|
||||
if stored="$(bao kv get -field=value ${lib.escapeShellArg controllerTokenStorePath} 2>"$err")" && [ -n "$stored" ]; then
|
||||
status="$(grafana "Bearer $stored" GET /api/prometheus/grafana/api/v1/alerts)"
|
||||
case "$status" in
|
||||
200) exit 0 ;;
|
||||
401) echo "Grafana refuses the stored token for ${controllerAccountName}; replacing it." ;;
|
||||
*) expect "reading alerts with the stored token" "$status" 200 ;;
|
||||
esac
|
||||
fi
|
||||
|
||||
status="$(grafana "$admin" GET "/api/serviceaccounts/$id/tokens")"
|
||||
expect "listing the ${controllerAccountName} tokens" "$status" 200
|
||||
for token_id in $(jq -r '.[].id' "$body"); do
|
||||
status="$(grafana "$admin" DELETE "/api/serviceaccounts/$id/tokens/$token_id")"
|
||||
expect "deleting token $token_id of ${controllerAccountName}" "$status" 200
|
||||
done
|
||||
|
||||
status="$(grafana "$admin" POST "/api/serviceaccounts/$id/tokens" \
|
||||
--data "{\"name\":\"${controllerAccountName}-$(date +%s)\"}")"
|
||||
expect "minting a ${controllerAccountName} token" "$status" 200
|
||||
key="$(jq -r .key "$body")"
|
||||
: >"$body"
|
||||
if ! printf '%s' "$key" | bao kv put ${lib.escapeShellArg controllerTokenStorePath} value=- >/dev/null 2>"$err"; then
|
||||
echo "could not store the new ${controllerAccountName} token at ${controllerTokenStorePath}; the next run replaces it." >&2
|
||||
cat "$err" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "stored a new ${controllerAccountName} token at ${controllerTokenStorePath}"
|
||||
'';
|
||||
};
|
||||
|
||||
systemd.timers.swarm-bao-grafana-controller-token = storeRefreshTimer "check swarm-controller's Grafana token and replace it if Grafana refuses it";
|
||||
|
||||
containers.${cfg.machine} = {
|
||||
autoStart = true;
|
||||
ephemeral = false;
|
||||
|
|
|
|||
|
|
@ -221,10 +221,10 @@ let
|
|||
&& builtins.elem "bao-client-key.pem:/var/lib/swarm-bao-pki/controller-key.pem" s.swarm-controller.serviceConfig.LoadCredential;
|
||||
}
|
||||
{
|
||||
# The forge token comes from the store, so the unit is handed no file for
|
||||
# it, and neither fixture runs the forge: a controller reaches it by name
|
||||
# wherever it runs.
|
||||
name = "the controller is handed no forge token file, and is pointed at the forge away from it";
|
||||
# The forge and Grafana tokens come from the store, so the unit is handed
|
||||
# no file for either, and neither fixture runs the forge or Grafana: a
|
||||
# controller reaches both by name wherever they run.
|
||||
name = "the controller is handed no forge token file, and is pointed at the forge and Grafana away from it";
|
||||
ok =
|
||||
lib.all
|
||||
(
|
||||
|
|
@ -235,6 +235,7 @@ let
|
|||
!(lib.any (c: lib.hasPrefix "forge-token:" c) u.serviceConfig.LoadCredential)
|
||||
&& !(u.environment ? SWARM_CONTROLLER_FORGE_TOKEN_FILE)
|
||||
&& u.environment.SWARM_CONTROLLER_FORGE_URL == "https://forge.t.local"
|
||||
&& u.environment.SWARM_CONTROLLER_GRAFANA_URL == "https://grafana.t.local"
|
||||
)
|
||||
[
|
||||
baoControllerHere
|
||||
|
|
|
|||
|
|
@ -597,8 +597,10 @@ let
|
|||
#
|
||||
# The admin password and the metrics bearer are the leaves it may create,
|
||||
# and without `update`: a grant that could replace one would let a second
|
||||
# run rotate a value Grafana or the metrics store already holds.
|
||||
name = "the Grafana reader reads its own client secret and creates its admin password and metrics bearer, and nothing else";
|
||||
# run rotate a value Grafana or the metrics store already holds. The
|
||||
# controller's Grafana token is the one leaf it may replace, which it
|
||||
# does when Grafana refuses the stored token.
|
||||
name = "the Grafana reader reads its own client secret, creates its admin password and metrics bearer, and writes the controller's Grafana token, and nothing else";
|
||||
ok =
|
||||
let
|
||||
s = baoGrantHere.systemd.services.swarm-bao-grafana-oidc-policy.script;
|
||||
|
|
@ -622,6 +624,14 @@ let
|
|||
"read"
|
||||
];
|
||||
}
|
||||
{
|
||||
path = "secret/data/swarm/controller/swarm-controller/grafana/token";
|
||||
caps = [
|
||||
"create"
|
||||
"read"
|
||||
"update"
|
||||
];
|
||||
}
|
||||
]
|
||||
&& !(lib.hasInfix "secret/data/swarm/agents" s)
|
||||
&& !(lib.hasInfix "secret/data/swarm/hives" s)
|
||||
|
|
@ -938,18 +948,27 @@ let
|
|||
&& !(noReaders ? swarm-bao-nats-callout-xkey);
|
||||
}
|
||||
{
|
||||
# The admin password is created under Grafana's leaf, so its unit waits
|
||||
# for the unit writing that leaf's role and policy; before it, the create
|
||||
# is refused. The second arm is the absence control.
|
||||
name = "grafana's admin password unit is ordered after grafana's policy unit";
|
||||
# The admin password and the controller's Grafana token are written under
|
||||
# Grafana's leaf, so their units wait for the unit writing that leaf's
|
||||
# role and policy; before it, the write is refused. The last arm is the
|
||||
# absence control.
|
||||
name = "grafana's admin password and controller token units are ordered after grafana's policy unit";
|
||||
ok =
|
||||
let
|
||||
u = baoGrantWithConsumers.systemd.services.swarm-bao-grafana-admin;
|
||||
in
|
||||
lib.elem "swarm-bao-grafana-oidc-policy.service" u.after
|
||||
&& lib.elem "swarm-bao-grafana-oidc-policy.service" u.wants
|
||||
&& !(lib.elem "swarm-bao-grafana-oidc-policy.service" u.requires)
|
||||
&& !(baoGrantNoReaders.systemd.services ? swarm-bao-grafana-admin);
|
||||
lib.all
|
||||
(
|
||||
unit:
|
||||
let
|
||||
u = baoGrantWithConsumers.systemd.services.${unit};
|
||||
in
|
||||
lib.elem "swarm-bao-grafana-oidc-policy.service" u.after
|
||||
&& lib.elem "swarm-bao-grafana-oidc-policy.service" u.wants
|
||||
&& !(lib.elem "swarm-bao-grafana-oidc-policy.service" u.requires)
|
||||
&& !(baoGrantNoReaders.systemd.services ? ${unit})
|
||||
)
|
||||
[
|
||||
"swarm-bao-grafana-admin"
|
||||
"swarm-bao-grafana-controller-token"
|
||||
];
|
||||
}
|
||||
{
|
||||
# The ordering is set apart from each reader's own definition, so it can
|
||||
|
|
@ -1100,6 +1119,7 @@ let
|
|||
"swarm-bao-forwarder-oidc"
|
||||
"swarm-bao-grafana-admin"
|
||||
"swarm-bao-grafana-metrics-bearer"
|
||||
"swarm-bao-grafana-controller-token"
|
||||
]
|
||||
++ policyReaders
|
||||
);
|
||||
|
|
@ -1715,6 +1735,25 @@ let
|
|||
"swarm-bao-forge-publisher-policy"
|
||||
];
|
||||
}
|
||||
{
|
||||
# The controller's Grafana token: read on the one leaf, pinned as the
|
||||
# whole stanza so an added write fails, and named by no policy but the
|
||||
# controller's and Grafana's, which writes it.
|
||||
name = "the controller reads its Grafana token, and only Grafana's grant writes it";
|
||||
ok =
|
||||
let
|
||||
s = baoGrantHere.systemd.services.swarm-bao-controller-policy.script;
|
||||
naming = lib.filter (
|
||||
u: lib.hasInfix "swarm-controller/grafana/token" baoGrantHere.systemd.services.${u}.script
|
||||
) grantingUnitNames;
|
||||
in
|
||||
lib.hasInfix "path \"secret/data/swarm/controller/swarm-controller/grafana/token\" {\n capabilities = [\"read\"]\n}" s
|
||||
&&
|
||||
lib.sort lib.lessThan naming == [
|
||||
"swarm-bao-controller-policy"
|
||||
"swarm-bao-grafana-oidc-policy"
|
||||
];
|
||||
}
|
||||
{
|
||||
# The forge's OIDC client secret by name: the forge publisher reads it,
|
||||
# and the secret publisher writes it under its `swarm/services/*` grant,
|
||||
|
|
|
|||
|
|
@ -249,6 +249,33 @@ let
|
|||
&& lib.hasInfix "admin reset-admin-password --password-from-stdin < /var/lib/grafana-admin/password" service.preStart
|
||||
&& lib.hasInfix "-config ${resetConfig}" service.script;
|
||||
}
|
||||
{
|
||||
# swarm-controller reads alerts with this token, so the account it
|
||||
# belongs to is created and kept a Viewer, the token lands at the path
|
||||
# the controller's grant reads, and an attempt against a Grafana or a
|
||||
# store that never answers ends and retries. The timer re-runs it, so
|
||||
# the unit must not stay active.
|
||||
name = "the controller's Grafana token is a Viewer account's, stored under Grafana's leaf, and each attempt is bounded";
|
||||
ok =
|
||||
let
|
||||
unit = grafanaRemoteAuthelia.systemd.services.swarm-bao-grafana-controller-token;
|
||||
s = unit.script;
|
||||
in
|
||||
!(lib.elem (toString (unit.serviceConfig.TimeoutStartSec or "infinity")) [
|
||||
"infinity"
|
||||
"0"
|
||||
])
|
||||
&& lib.hasInfix "curl --silent --show-error --connect-timeout 5 --max-time 15" s
|
||||
&& lib.hasInfix "--data '{\"name\":\"swarm-controller\",\"role\":\"Viewer\",\"isDisabled\":false}'" s
|
||||
&& lib.hasInfix "--data '{\"role\":\"Viewer\",\"isDisabled\":false}'" s
|
||||
&& lib.hasInfix "bao kv put secret/swarm/controller/swarm-controller/grafana/token value=-" s
|
||||
&& lib.hasInfix "--unix-socket /run/swarm-grafana/grafana.sock" s
|
||||
&& unit.environment.BAO_CLIENT_CERT == "/etc/pki/bao-grafana-oidc.pem"
|
||||
&& unit.serviceConfig.Restart == "on-failure"
|
||||
&& !(unit.serviceConfig.RemainAfterExit or false)
|
||||
&& lib.elem "swarm-bao-grafana-admin.service" unit.after
|
||||
&& grafanaRemoteAuthelia.systemd.timers ? swarm-bao-grafana-controller-token;
|
||||
}
|
||||
{
|
||||
# Same 403-not-a-miss reason as the matrix and queue arms below: the
|
||||
# reader's grant covers the `services` prefix, so a path outside it is
|
||||
|
|
|
|||
|
|
@ -777,6 +777,7 @@ let
|
|||
"swarm-bao-grafana-oidc"
|
||||
"swarm-bao-grafana-admin"
|
||||
"swarm-bao-grafana-metrics-bearer"
|
||||
"swarm-bao-grafana-controller-token"
|
||||
"swarm-bao-nats-callout-xkey"
|
||||
"swarm-bao-ci-runner"
|
||||
"swarm-bao-otel-oidc"
|
||||
|
|
@ -799,7 +800,7 @@ let
|
|||
!(lib.any (n: lib.elem n (s.after ++ s.wants ++ s.requires)) storeUnits);
|
||||
in
|
||||
lib.all waits readers
|
||||
&& lib.length readersElsewhere == 7
|
||||
&& lib.length readersElsewhere == 8
|
||||
&& lib.all namesNoStoreUnit readersElsewhere
|
||||
&& !(lib.any (u: storeOnlyNoHive.systemd.services ? ${u}) readers);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -73,6 +73,9 @@ The swarm's control plane. The swarm UI and `swarmctl` are its clients.
|
|||
[`docs/swarm/review-classifier.md`](../docs/swarm/review-classifier.md#swarm-level-settings).
|
||||
- **Relays** — each agent's terminal and turn-state header as SSE, agent
|
||||
icons, the cross-repo issue report, and the UI's quick links.
|
||||
- **Alerts** — the swarm Grafana's current alerts (`GET /api/alerts`), read
|
||||
at request time with a Viewer service-account token from the secret store
|
||||
→ [`docs/swarm/secrets.md`](../docs/swarm/secrets.md).
|
||||
|
||||
It reads its configuration once at startup, from the environment the nix module
|
||||
sets. It persists `webhook-secret` and `operator-matrix-admins.json` in its
|
||||
|
|
|
|||
|
|
@ -508,6 +508,7 @@ pub(crate) mod tests {
|
|||
reviews: Arc::default(),
|
||||
try_outs: crate::review_classifier::TryOuts::default(),
|
||||
runs: crate::review_classifier::runs::Runs::Live(None),
|
||||
alerts: None,
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
391
swarm-controller/src/alerts.rs
Normal file
391
swarm-controller/src/alerts.rs
Normal file
|
|
@ -0,0 +1,391 @@
|
|||
//! `GET /api/alerts`: the alerts the swarm's Grafana holds right now, read at
|
||||
//! request time for swarm-ui's alerts page.
|
||||
//!
|
||||
//! Grafana's URL comes from `swarm-controller.nix`
|
||||
//! (`SWARM_CONTROLLER_GRAFANA_URL`). The token is a Viewer service account's,
|
||||
//! read from the swarm's secret store at
|
||||
//! `swarm_secret_client::grafana::controller_token_path` under this daemon's
|
||||
//! own certificate. `swarm-bao-grafana-controller-token`, on Grafana's host,
|
||||
//! mints it and replaces it whenever Grafana refuses the stored one, so a 401
|
||||
//! here re-reads the store once before giving up.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::future::Future;
|
||||
use std::pin::Pin;
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
|
||||
use anyhow::{Context, Result};
|
||||
use axum::Json;
|
||||
use axum::extract::State;
|
||||
use axum::http::StatusCode;
|
||||
use problem_details::ProblemDetails;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use utoipa::ToSchema;
|
||||
|
||||
use crate::{AppState, error_problem};
|
||||
|
||||
/// The env var `swarm-controller.nix` sets.
|
||||
const URL_ENV: &str = "SWARM_CONTROLLER_GRAFANA_URL";
|
||||
|
||||
/// Grafana's Prometheus-compatible list of alert instances, relative to its
|
||||
/// base URL. Instances in the `Normal` state are not listed.
|
||||
const ALERTS_PATH: &str = "api/prometheus/grafana/api/v1/alerts";
|
||||
|
||||
/// Bound on reaching Grafana.
|
||||
const CONNECT_TIMEOUT: Duration = Duration::from_secs(5);
|
||||
/// Bound on one whole request to Grafana, body included.
|
||||
const TIMEOUT: Duration = Duration::from_secs(15);
|
||||
|
||||
type TokenFuture = Pin<Box<dyn Future<Output = Result<String>> + Send>>;
|
||||
|
||||
/// Where the token comes from: the store in the daemon, a stand-in in tests.
|
||||
pub(crate) type TokenSource = Arc<dyn Fn() -> TokenFuture + Send + Sync>;
|
||||
|
||||
/// One alert instance, as `GET /api/alerts` serves it.
|
||||
#[derive(Clone, Debug, PartialEq, Serialize, ToSchema)]
|
||||
pub(crate) struct Alert {
|
||||
/// The rule's title (Grafana's `alertname` label).
|
||||
pub name: String,
|
||||
/// Grafana's state for the instance: `Alerting`, `Pending`, `NoData` or
|
||||
/// `Error`.
|
||||
pub state: String,
|
||||
/// When the instance entered its state, RFC 3339, as Grafana reports it.
|
||||
pub active_at: Option<String>,
|
||||
/// The rule's `summary` annotation, templated by Grafana.
|
||||
pub summary: Option<String>,
|
||||
/// Every label of the instance, as Grafana reports them.
|
||||
pub labels: BTreeMap<String, String>,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct PromResponse {
|
||||
data: PromData,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct PromData {
|
||||
alerts: Vec<PromAlert>,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct PromAlert {
|
||||
#[serde(default)]
|
||||
labels: BTreeMap<String, String>,
|
||||
#[serde(default)]
|
||||
annotations: BTreeMap<String, String>,
|
||||
state: String,
|
||||
#[serde(rename = "activeAt")]
|
||||
active_at: Option<String>,
|
||||
}
|
||||
|
||||
impl From<PromAlert> for Alert {
|
||||
fn from(a: PromAlert) -> Self {
|
||||
Self {
|
||||
name: a.labels.get("alertname").cloned().unwrap_or_default(),
|
||||
state: a.state,
|
||||
active_at: a.active_at,
|
||||
summary: a.annotations.get("summary").cloned(),
|
||||
labels: a.labels,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Grafana's alerts API, with the token last read from the store.
|
||||
pub(crate) struct Grafana {
|
||||
http: reqwest::Client,
|
||||
alerts_url: url::Url,
|
||||
source: TokenSource,
|
||||
token: tokio::sync::Mutex<Option<String>>,
|
||||
}
|
||||
|
||||
impl Grafana {
|
||||
/// # Errors
|
||||
/// A base URL the alerts path cannot be joined onto, or a client that
|
||||
/// cannot be built.
|
||||
pub(crate) fn new(base: &url::Url, source: TokenSource, timeout: Duration) -> Result<Self> {
|
||||
let http = reqwest::Client::builder()
|
||||
.connect_timeout(CONNECT_TIMEOUT)
|
||||
.timeout(timeout)
|
||||
.build()
|
||||
.context("building the Grafana HTTP client")?;
|
||||
Ok(Self {
|
||||
http,
|
||||
alerts_url: base
|
||||
.join(ALERTS_PATH)
|
||||
.with_context(|| format!("joining {ALERTS_PATH} onto {base}"))?,
|
||||
source,
|
||||
token: tokio::sync::Mutex::new(None),
|
||||
})
|
||||
}
|
||||
|
||||
/// The held token, or a fresh one from the source when `reread` or none is
|
||||
/// held yet.
|
||||
async fn token(&self, reread: bool) -> Result<String> {
|
||||
let mut held = self.token.lock().await;
|
||||
if let Some(t) = held.as_ref().filter(|_| !reread) {
|
||||
return Ok(t.clone());
|
||||
}
|
||||
let fresh = (self.source)().await?;
|
||||
*held = Some(fresh.clone());
|
||||
Ok(fresh)
|
||||
}
|
||||
|
||||
async fn get(&self, token: &str) -> Result<reqwest::Response> {
|
||||
self.http
|
||||
.get(self.alerts_url.clone())
|
||||
.bearer_auth(token)
|
||||
.send()
|
||||
.await
|
||||
.with_context(|| format!("GET {}", self.alerts_url))
|
||||
}
|
||||
|
||||
/// Every alert instance Grafana lists now.
|
||||
///
|
||||
/// # Errors
|
||||
/// No token in the store, Grafana unreachable or slower than the client's
|
||||
/// timeout, a 401 for both the held token and a re-read one, any other
|
||||
/// non-success status, or a body that is not the alerts shape.
|
||||
pub(crate) async fn alerts(&self) -> Result<Vec<Alert>> {
|
||||
let mut res = self.get(&self.token(false).await?).await?;
|
||||
if res.status() == reqwest::StatusCode::UNAUTHORIZED {
|
||||
res = self.get(&self.token(true).await?).await?;
|
||||
}
|
||||
let status = res.status();
|
||||
if !status.is_success() {
|
||||
anyhow::bail!("Grafana answered {status} to GET {}", self.alerts_url);
|
||||
}
|
||||
let body: PromResponse = res
|
||||
.json()
|
||||
.await
|
||||
.with_context(|| format!("reading Grafana's answer to GET {}", self.alerts_url))?;
|
||||
Ok(body.data.alerts.into_iter().map(Alert::from).collect())
|
||||
}
|
||||
}
|
||||
|
||||
/// One login and one read. An empty value is refused like an absent one.
|
||||
async fn token_from_store() -> Result<String> {
|
||||
let path = swarm_secret_client::grafana::controller_token_path()?;
|
||||
let store = crate::store::connect()
|
||||
.await
|
||||
.context("logging in to the swarm secret store")?;
|
||||
let stored: Option<swarm_secret_client::grafana::ControllerToken> = store
|
||||
.read_optional(&path)
|
||||
.await
|
||||
.with_context(|| format!("reading {path}"))?;
|
||||
match stored {
|
||||
Some(t) if !t.value.trim().is_empty() => Ok(t.value.trim().to_owned()),
|
||||
_ => anyhow::bail!(
|
||||
"nothing at {path}: swarm-bao-grafana-controller-token, on the host that runs \
|
||||
Grafana, has not stored this daemon's Grafana token"
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
/// The daemon's Grafana client, or `None` when `SWARM_CONTROLLER_GRAFANA_URL`
|
||||
/// is unset or unusable, which is logged.
|
||||
pub(crate) fn from_env() -> Option<Arc<Grafana>> {
|
||||
let raw = std::env::var(URL_ENV).ok()?;
|
||||
let built = url::Url::parse(&raw)
|
||||
.with_context(|| format!("parse {URL_ENV} ({raw}) as a URL"))
|
||||
.and_then(|base| {
|
||||
let source: TokenSource = Arc::new(|| Box::pin(token_from_store()));
|
||||
Grafana::new(&base, source, TIMEOUT)
|
||||
});
|
||||
match built {
|
||||
Ok(g) => Some(Arc::new(g)),
|
||||
Err(e) => {
|
||||
tracing::warn!(error = %format!("{e:#}"), "Grafana misconfigured; /api/alerts is off");
|
||||
None
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The alerts the swarm's Grafana holds now: firing, pending, and those whose
|
||||
/// rule has no data or failed to evaluate.
|
||||
#[utoipa::path(
|
||||
get,
|
||||
path = "/api/alerts",
|
||||
responses(
|
||||
(status = 200, description = "every alert instance Grafana lists now", body = Vec<Alert>),
|
||||
(status = 503, description = "no Grafana is configured here, its token is not in the store \
|
||||
yet, or Grafana refused, failed or did not answer in time (problem+json)", body = String),
|
||||
),
|
||||
tag = "alerts"
|
||||
)]
|
||||
pub(crate) async fn get_alerts(
|
||||
State(state): State<AppState>,
|
||||
) -> Result<Json<Vec<Alert>>, ProblemDetails> {
|
||||
let Some(grafana) = state.alerts.as_ref() else {
|
||||
return Err(error_problem(
|
||||
StatusCode::SERVICE_UNAVAILABLE,
|
||||
"no Grafana is configured on this host",
|
||||
));
|
||||
};
|
||||
grafana.alerts().await.map(Json).map_err(|e| {
|
||||
let detail = format!("{e:#}");
|
||||
tracing::warn!(error = %detail, "reading Grafana's alerts failed");
|
||||
error_problem(StatusCode::SERVICE_UNAVAILABLE, &detail)
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use axum::body::Body;
|
||||
use axum::http::{HeaderMap, Request};
|
||||
use std::sync::Mutex;
|
||||
use tower::ServiceExt as _;
|
||||
|
||||
/// What the stand-in Grafana answers to a request with the accepted token.
|
||||
const ALERTS_BODY: &str = r#"{"status":"success","data":{"alerts":[{
|
||||
"labels":{"alertname":"swarm queue connect failed","hive":"pr1ma","agent":"atlas"},
|
||||
"annotations":{"summary":"pr1ma/atlas could not connect"},
|
||||
"state":"Alerting","activeAt":"2026-10-15T09:00:00Z","value":"1e+00"}]}}"#;
|
||||
|
||||
/// A stand-in Grafana on a loopback port: the alerts path answers
|
||||
/// [`ALERTS_BODY`] to `Bearer <accepts>` and 401 to anything else, after
|
||||
/// `delay`.
|
||||
async fn fake_grafana(accepts: &'static str, delay: Duration) -> url::Url {
|
||||
let app = axum::Router::new().route(
|
||||
&format!("/{ALERTS_PATH}"),
|
||||
axum::routing::get(move |headers: HeaderMap| async move {
|
||||
tokio::time::sleep(delay).await;
|
||||
let ok = headers
|
||||
.get(axum::http::header::AUTHORIZATION)
|
||||
.is_some_and(|v| v == format!("Bearer {accepts}").as_str());
|
||||
if ok {
|
||||
(StatusCode::OK, ALERTS_BODY)
|
||||
} else {
|
||||
(StatusCode::UNAUTHORIZED, "")
|
||||
}
|
||||
}),
|
||||
);
|
||||
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let addr = listener.local_addr().unwrap();
|
||||
tokio::spawn(async move { axum::serve(listener, app).await });
|
||||
url::Url::parse(&format!("http://{addr}/")).unwrap()
|
||||
}
|
||||
|
||||
/// A token source handing out `tokens` in order, the last one forever,
|
||||
/// counting its calls.
|
||||
fn tokens(tokens: &'static [&'static str]) -> (TokenSource, Arc<Mutex<usize>>) {
|
||||
let calls = Arc::new(Mutex::new(0));
|
||||
let counted = Arc::clone(&calls);
|
||||
let source: TokenSource = Arc::new(move || {
|
||||
let mut n = counted.lock().unwrap();
|
||||
let t = tokens[(*n).min(tokens.len() - 1)];
|
||||
*n += 1;
|
||||
Box::pin(async move { Ok(t.to_owned()) })
|
||||
});
|
||||
(source, calls)
|
||||
}
|
||||
|
||||
async fn get_alerts_with(grafana: Option<Grafana>) -> (StatusCode, serde_json::Value) {
|
||||
let (mut state, _) = crate::tests::state_with_roster();
|
||||
state.alerts = grafana.map(Arc::new);
|
||||
let req = Request::get("/api/alerts")
|
||||
.body(Body::empty())
|
||||
.expect("a request");
|
||||
let res = crate::build_app(state)
|
||||
.oneshot(req)
|
||||
.await
|
||||
.expect("infallible");
|
||||
let status = res.status();
|
||||
let bytes = axum::body::to_bytes(res.into_body(), usize::MAX)
|
||||
.await
|
||||
.expect("a body");
|
||||
(status, serde_json::from_slice(&bytes).unwrap_or_default())
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn serves_grafanas_alerts_read_with_the_stored_token() {
|
||||
let base = fake_grafana("viewer-token", Duration::ZERO).await;
|
||||
let (source, calls) = tokens(&["viewer-token"]);
|
||||
let grafana = Grafana::new(&base, source, TIMEOUT).unwrap();
|
||||
let (status, body) = get_alerts_with(Some(grafana)).await;
|
||||
assert_eq!(status, StatusCode::OK, "{body}");
|
||||
assert_eq!(
|
||||
body,
|
||||
serde_json::json!([{
|
||||
"name": "swarm queue connect failed",
|
||||
"state": "Alerting",
|
||||
"active_at": "2026-10-15T09:00:00Z",
|
||||
"summary": "pr1ma/atlas could not connect",
|
||||
"labels": {
|
||||
"alertname": "swarm queue connect failed",
|
||||
"hive": "pr1ma",
|
||||
"agent": "atlas",
|
||||
},
|
||||
}])
|
||||
);
|
||||
assert_eq!(*calls.lock().unwrap(), 1);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn the_held_token_is_reused_across_requests() {
|
||||
let base = fake_grafana("viewer-token", Duration::ZERO).await;
|
||||
let (source, calls) = tokens(&["viewer-token"]);
|
||||
let grafana = Grafana::new(&base, source, TIMEOUT).unwrap();
|
||||
grafana.alerts().await.unwrap();
|
||||
grafana.alerts().await.unwrap();
|
||||
assert_eq!(*calls.lock().unwrap(), 1);
|
||||
}
|
||||
|
||||
/// The minting unit replaces a token Grafana refuses, so a 401 re-reads
|
||||
/// the store and sends the request once more.
|
||||
#[tokio::test]
|
||||
async fn a_refused_token_is_read_again_from_the_store() {
|
||||
let base = fake_grafana("replaced", Duration::ZERO).await;
|
||||
let (source, calls) = tokens(&["stale", "replaced"]);
|
||||
let grafana = Grafana::new(&base, source, TIMEOUT).unwrap();
|
||||
let alerts = grafana.alerts().await.unwrap();
|
||||
assert_eq!(alerts.len(), 1);
|
||||
assert_eq!(*calls.lock().unwrap(), 2);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_token_refused_twice_is_a_503_naming_the_status() {
|
||||
let base = fake_grafana("replaced", Duration::ZERO).await;
|
||||
let (source, calls) = tokens(&["stale"]);
|
||||
let grafana = Grafana::new(&base, source, TIMEOUT).unwrap();
|
||||
let (status, body) = get_alerts_with(Some(grafana)).await;
|
||||
assert_eq!(status, StatusCode::SERVICE_UNAVAILABLE);
|
||||
assert!(body["detail"].as_str().unwrap().contains("401"), "{body}");
|
||||
assert_eq!(*calls.lock().unwrap(), 2);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_grafana_slower_than_the_timeout_is_a_503() {
|
||||
let base = fake_grafana("viewer-token", Duration::from_secs(5)).await;
|
||||
let (source, _) = tokens(&["viewer-token"]);
|
||||
let grafana = Grafana::new(&base, source, Duration::from_millis(200)).unwrap();
|
||||
let started = std::time::Instant::now();
|
||||
let (status, _) = get_alerts_with(Some(grafana)).await;
|
||||
assert_eq!(status, StatusCode::SERVICE_UNAVAILABLE);
|
||||
assert!(started.elapsed() < Duration::from_secs(4));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn no_token_in_the_store_is_a_503_saying_so() {
|
||||
let base = fake_grafana("viewer-token", Duration::ZERO).await;
|
||||
let source: TokenSource =
|
||||
Arc::new(|| Box::pin(async { anyhow::bail!("nothing at the token path") }));
|
||||
let grafana = Grafana::new(&base, source, TIMEOUT).unwrap();
|
||||
let (status, body) = get_alerts_with(Some(grafana)).await;
|
||||
assert_eq!(status, StatusCode::SERVICE_UNAVAILABLE);
|
||||
assert!(
|
||||
body["detail"]
|
||||
.as_str()
|
||||
.unwrap()
|
||||
.contains("nothing at the token path"),
|
||||
"{body}"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn no_grafana_configured_is_a_503() {
|
||||
let (status, _) = get_alerts_with(None).await;
|
||||
assert_eq!(status, StatusCode::SERVICE_UNAVAILABLE);
|
||||
}
|
||||
}
|
||||
|
|
@ -168,6 +168,7 @@ mod tests {
|
|||
try_outs: super::super::review_classifier::TryOuts::default(),
|
||||
reviews: std::sync::Arc::default(),
|
||||
runs: crate::review_classifier::runs::Runs::Live(None),
|
||||
alerts: None,
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -138,6 +138,7 @@ mod tests {
|
|||
try_outs: super::super::review_classifier::TryOuts::default(),
|
||||
reviews: std::sync::Arc::default(),
|
||||
runs: crate::review_classifier::runs::Runs::Live(None),
|
||||
alerts: None,
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -973,6 +973,7 @@ pub(crate) mod tests {
|
|||
try_outs: super::super::review_classifier::TryOuts::default(),
|
||||
reviews: std::sync::Arc::default(),
|
||||
runs: crate::review_classifier::runs::Runs::Live(None),
|
||||
alerts: None,
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -50,6 +50,7 @@ mod agent_login;
|
|||
mod agent_renewal;
|
||||
mod agent_state_stream;
|
||||
mod agent_status;
|
||||
mod alerts;
|
||||
mod auth;
|
||||
mod builder;
|
||||
mod claude_login;
|
||||
|
|
@ -839,6 +840,7 @@ fn socket_path() -> PathBuf {
|
|||
(name = "schedules", description = "scheduled prompts (`swarmctl schedule`)"),
|
||||
(name = "review-classifier", description = "trying review classifications against a PR (swarm-ui's try-out page), and the classifier's swarm-wide settings"),
|
||||
(name = "pulls", description = "open pull requests, for the try-out page's dropdowns"),
|
||||
(name = "alerts", description = "the swarm Grafana's current alerts (swarm-ui's alerts page)"),
|
||||
)
|
||||
)]
|
||||
struct ApiDoc;
|
||||
|
|
@ -972,6 +974,9 @@ struct AppState {
|
|||
reviews: Arc<review_classifier::review::Reviews>,
|
||||
/// The review classifier's finished runs.
|
||||
runs: review_classifier::runs::Runs,
|
||||
/// Grafana's alerts API, for `GET /api/alerts`. `None` when no Grafana URL
|
||||
/// is configured.
|
||||
alerts: Option<Arc<alerts::Grafana>>,
|
||||
}
|
||||
|
||||
/// Env var the controller's NixOS module sets from
|
||||
|
|
@ -3451,6 +3456,7 @@ async fn main() -> Result<()> {
|
|||
try_outs: review_classifier::TryOuts::default(),
|
||||
reviews: Arc::default(),
|
||||
runs,
|
||||
alerts: alerts::from_env(),
|
||||
};
|
||||
|
||||
let app = build_app(state);
|
||||
|
|
@ -3468,6 +3474,7 @@ async fn main() -> Result<()> {
|
|||
fn build_app(state: AppState) -> axum::Router {
|
||||
let (router, api) = OpenApiRouter::<AppState>::with_openapi(ApiDoc::openapi())
|
||||
.routes(routes!(health))
|
||||
.routes(routes!(alerts::get_alerts))
|
||||
.routes(routes!(get_hives))
|
||||
.routes(routes!(get_hives_status))
|
||||
.routes(routes!(get_links))
|
||||
|
|
@ -3660,6 +3667,7 @@ mod tests {
|
|||
try_outs: super::review_classifier::TryOuts::default(),
|
||||
reviews: std::sync::Arc::default(),
|
||||
runs: super::review_classifier::runs::Runs::Live(None),
|
||||
alerts: None,
|
||||
};
|
||||
(state, sched)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -585,6 +585,7 @@ mod tests {
|
|||
try_outs: super::super::review_classifier::TryOuts::default(),
|
||||
reviews: std::sync::Arc::default(),
|
||||
runs: crate::review_classifier::runs::Runs::Live(None),
|
||||
alerts: None,
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
54
swarm-secret-client/src/grafana.rs
Normal file
54
swarm-secret-client/src/grafana.rs
Normal file
|
|
@ -0,0 +1,54 @@
|
|||
//! The Grafana agreement: where `swarm-controller`'s token for Grafana's API
|
||||
//! lives, and what the object at that path holds.
|
||||
//!
|
||||
//! `swarm-bao-grafana-controller-token`, on the host that runs Grafana, mints
|
||||
//! the token for a Viewer service account and writes it here under Grafana's
|
||||
//! own certificate; `swarm-controller` reads it under its own. Those two
|
||||
//! principals' stanzas are the only ones in the store that cover this path.
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use crate::{
|
||||
Error,
|
||||
path::{Kind, principal_prefix},
|
||||
};
|
||||
|
||||
/// The path holding `swarm-controller`'s Grafana token.
|
||||
///
|
||||
/// Under [`Kind::Controller`], which no hive's policy reads. The nix half is
|
||||
/// `swarm-bao.nix`'s `controllerGrafanaTokenLeaf`.
|
||||
///
|
||||
/// # Errors
|
||||
/// Never in practice: the name segment is a constant. The `Result` is
|
||||
/// [`principal_prefix`]'s.
|
||||
pub fn controller_token_path() -> Result<String, Error> {
|
||||
let prefix = principal_prefix(Kind::Controller, crate::matrix::CONTROLLER)?;
|
||||
Ok(format!("{prefix}/grafana/token"))
|
||||
}
|
||||
|
||||
/// What [`controller_token_path`] holds.
|
||||
///
|
||||
/// No `Debug` derive: `value` reads every alert and dashboard in the swarm's
|
||||
/// Grafana, and a derived `Debug` is one `{:?}` away from a log line.
|
||||
#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct ControllerToken {
|
||||
/// The token itself. The minting unit writes it with
|
||||
/// `bao kv put … value=-`, so this name is load-bearing for a writer this
|
||||
/// crate does not control.
|
||||
pub value: String,
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn the_controllers_grafana_token_is_under_the_controllers_prefix() {
|
||||
// The literal is what `swarm-bao.nix` grants Grafana's role and the
|
||||
// controller's, and what the minting unit writes.
|
||||
assert_eq!(
|
||||
controller_token_path().expect("a constant segment is legal"),
|
||||
"swarm/controller/swarm-controller/grafana/token"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
@ -5,10 +5,10 @@
|
|||
//! the rules every path obeys ([`path`]), the translation from this
|
||||
//! deployment's environment into a logged-in client ([`client`]), and, per kind
|
||||
//! of secret, the path it lives at together with the fields it holds
|
||||
//! ([`matrix`], [`queue`], [`mtls`], [`forge`], [`github`], [`acp`], [`ci`],
|
||||
//! [`review_classifier`], [`builder`], [`claude_login`]). Each of those is a
|
||||
//! thing the controller and a hive must say identically, so it is said once
|
||||
//! here.
|
||||
//! ([`matrix`], [`queue`], [`mtls`], [`forge`], [`github`], [`grafana`],
|
||||
//! [`acp`], [`ci`], [`review_classifier`], [`builder`], [`claude_login`]).
|
||||
//! Each of those is a thing the controller and a hive must say identically, so
|
||||
//! it is said once here.
|
||||
//!
|
||||
//! [`policy`] is the same kind of agreement seen from the other side: which of
|
||||
//! those paths a given principal's own token may read — a hive's, and an
|
||||
|
|
@ -31,6 +31,7 @@ pub mod claude_login;
|
|||
pub mod client;
|
||||
pub mod forge;
|
||||
pub mod github;
|
||||
pub mod grafana;
|
||||
pub mod matrix;
|
||||
pub mod mtls;
|
||||
pub mod path;
|
||||
|
|
|
|||
Loading…
Reference in a new issue