swarm UI: an alerts page, read by the controller with a Viewer Grafana token from bao

swarm-bao-grafana-controller-token, on the host that runs Grafana, logs in
to Grafana's API over its socket as the built-in admin (the password
swarm-bao-grafana-admin minted, #5138) and keeps a `swarm-controller`
service account that is an enabled Viewer. It mints a token for that
account and writes it to bao at
swarm/controller/swarm-controller/grafana/token under Grafana's leaf. A
ten-minute timer runs it again. The unit keeps a stored token Grafana
accepts. If Grafana refuses the stored token with 401, or bao holds
none, the unit deletes every token of the account and mints a new one.
Grafana's grant gains create/read/update on that leaf, and the
controller's gains read.

swarm-controller serves GET /api/alerts from Grafana's
/api/prometheus/grafana/api/v1/alerts. It reads the token from bao,
keeps it in memory, and re-reads bao once on a 401. The HTTP client has
5s connect and 15s request timeouts. Any failure is a 503 that names its
cause. swarm-ui gets an /alerts page listing them, built the same way as
the hives page.

This is option (c) on #5129, the operator's choice.

nix/host-modules/swarm-bao.nix

@@ -369,9 +369,9 @@ let
   # Each hive's matrix sender token (`matrix_account::hive_sender`) grants `read`
   # for the same keep-if-live reason, and `+` for the same glob-only-as-last-segment reason.
   #
-  # Three leaves it reads and never writes: the swarm appservice token
-  # (matrix-ctl, `matrixCtlPolicyText` below), its own OIDC client secret (the
-  # publisher) and its own forge token (the forge's host, `forgePublisherWriters`).
+  # Four leaves it reads and never writes: the swarm appservice token (matrix-ctl),
+  # its OIDC client secret (the publisher), its forge token (the forge's host)
+  # and its Grafana token (Grafana's host). Each writer's grant is named below.
   #
   # The agent PKI grant is its only path on that mount: it can ask the one role
   # for a certificate, not write that role or reach the issuer.
@@ -457,6 +457,10 @@ let
       capabilities = ["read"]
     }
 
+    path "${credentialMountPath}/data/${controllerGrafanaTokenLeaf}" {
+      capabilities = ["read"]
+    }
+
     path "${agentPkiMountPath}/issue/${agentPkiRoleName}" {
       capabilities = ["update"]
     }
@@ -582,6 +586,11 @@ let
   # it; the controller reads it and holds no other copy.
   controllerForgeTokenLeaf = "swarm/controller/swarm-controller/forge/token";
 
+  # The controller's Viewer token for Grafana's API, the nix half of
+  # `swarm_secret_client::grafana::controller_token_path`. Grafana's host
+  # writes it; the controller reads it and holds no other copy.
+  controllerGrafanaTokenLeaf = "swarm/controller/swarm-controller/grafana/token";
+
   # The forge's runner registration token, the nix half of
   # `swarm_secret_client::ci::registration_token_path`. The controller writes
   # it and cannot read it back (`forge::runner_token` keeps the value it last
@@ -911,6 +920,10 @@ let
   # in ./swarm-grafana.nix) and the bearer its metrics datasource presents
   # (`swarm-bao-grafana-metrics-bearer`). `create` without `update`, so a stored
   # value is never replaced.
+  #
+  # The controller's Grafana token is the one it may replace:
+  # `swarm-bao-grafana-controller-token` mints a new one when Grafana refuses
+  # the stored one, as it does after its database is recreated.
   grafanaOidcReaders = [
     {
       name = "swarm-grafana-oidc";
@@ -918,7 +931,12 @@ let
       policyText =
         readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.grafana.oidc.clientId}/oidc/client"
         + createStanza "${credentialMountPath}/data/${grafanaAdminLeaf}"
-        + createStanza "${credentialMountPath}/data/${grafanaMetricsBearerLeaf}";
+        + createStanza "${credentialMountPath}/data/${grafanaMetricsBearerLeaf}"
+        + ''
+          path "${credentialMountPath}/data/${controllerGrafanaTokenLeaf}" {
+            capabilities = ["create", "read", "update"]
+          }
+        '';
     }
   ];