notify router: deliveries log, dry-run hit counts, api and page

Cargo.lock

@@ -5054,6 +5054,7 @@ dependencies = [
  "swarm-queue-client",
  "swarm-secret-client",
  "swarm-wire",
+ "tempfile",
  "tokio",
  "tracing",
 ]

docs/swarm/notify-router.md

@@ -171,17 +171,68 @@ controller that predates it, gets no answer to the request and logs that. A cont
 router runs the built-ins as before. A router that predates this ignores the
 document, so the page and the API show a state the router isn't yet running.
 
+## Deliveries and dry runs
+
+**The deliveries log.** The router appends one row to the `NOTIFY_ROUTER_LOG`
+stream for each delivery it makes, and the stream keeps the newest 500. A
+delivery is a todo op queued for an agent or a review requested from the
+controller, whether it went through or not. A row has:
+
+| Field                     | Holds                                                                              |
+| ------------------------- | ---------------------------------------------------------------------------------- |
+| `at`                      | Unix seconds                                                                       |
+| `flow`, `node`            | the flow and the node that delivered                                               |
+| `event`, `repo`, `number` | the forge event type and the issue or PR it concerns                               |
+| `kind`, `target`          | `todo` and the agent, or `request_reviewer` and the reviewer                       |
+| `ok`, `detail`            | whether it went through, and the op and key, the controller's answer, or the error |
+
+A review the controller skipped (`skipped: <why>`) is `ok`: nothing was
+wrong. An event the router retries after a failed step writes its rows again,
+so a retry shows twice. When the router fails to write a row, it logs the failure and the delivery
+stands.
+
+**The dry run.** The router runs a flows document over the newest 50 events of
+the `SWARM_EVENTS` stream and returns, for each flow the document turns on,
+how many of those events reached each node. A `match` node counts the events
+that passed it. It uses the walk the router delivers with and delivers nothing:
+no todo op, no review request, no log row and no snapshot written. It keeps
+its own per-issue snapshots, starting empty, so the first assign event it
+replays for an issue counts every assignee as newly added. Every recipient
+counts as an agent.
+
+**The API.** The router answers two NATS requests from swarm-controller, which
+serves them next to `GET /api/notify/flows`:
+
+| Route                        | Router subject                 | Answers                                                            |
+| ---------------------------- | ------------------------------ | ------------------------------------------------------------------ |
+| `GET /api/notify/deliveries` | `$SWARM.router.api.deliveries` | the newest rows, newest first; `?limit=` up to 500, 100 by default |
+| `GET /api/notify/dry-run`    | `$SWARM.router.api.dry-run`    | the saved flows document, run over the newest 50 events            |
+
+The dry-run request carries a whole document, so a draft runs the same way.
+The controller sends the saved one. [`/notifications`](ui.md#notification-flows)
+shows both: the recent deliveries, and the counts beside each node.
+
+**Deploying in any order.** A router that predates these subjects doesn't
+subscribe to them, and the queue answers the controller's request at once with
+no responders. Both routes then answer 503 saying the router doesn't answer
+and may be an older build, and the page shows that in place of the table and
+the counts. A request the router hears and doesn't answer within 10 seconds is
+the same 503. A router that answers with a refusal, or with something the
+controller can't read, is 502. A controller that predates the routes has no
+route: the page, if newer, shows the failure.
+
 ## Credentials and grants
 
-| Principal                               | Grant                                                                                                                                                                                                                                                                                                                                                                                 |
+| Principal                               | Grant                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
-| --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| --------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `swarm-notify-router`, in its container | store: `read` on `swarm/services/<router client id>/oidc/client`. Queue: its own consumer on `SWARM_EVENTS` (create, pull, ack), publish on `$SWARM.router.todo.*` and on `$SWARM.router.sink.request-reviewer`, create `ROUTER_TODOS`, its own `notify-router-state` bucket, read on the `notify-flows` bucket, and `STREAM.INFO` on the agent-status bucket, which lists the agents |
+| `swarm-notify-router`, in its container | store: `read` on `swarm/services/<router client id>/oidc/client`. Queue: its own consumer on `SWARM_EVENTS` (create, pull, ack) and `STREAM.MSG.GET` on it for the dry run, publish on `$SWARM.router.todo.*` and on `$SWARM.router.sink.request-reviewer`, create `ROUTER_TODOS`, create `NOTIFY_ROUTER_LOG`, publish its one subject and read it by sequence, its own `notify-router-state` bucket, read on the `notify-flows` bucket, `STREAM.INFO` on the agent-status bucket, which lists the agents, and subscribe to the two `$SWARM.router.api.*` request subjects and answer each once, for 10 seconds |
-| swarm-controller                        | publish on `$SWARM.events.forge.*`; subscribe to `$SWARM.router.sink.request-reviewer` and answer each request once, for 30 seconds; creates and writes the `notify-flows` bucket (every bucket, by its `$KV.*.>` grant)                                                                                                                                                              |
+| swarm-controller                        | publish on `$SWARM.events.forge.*` and on the two `$SWARM.router.api.*` request subjects; subscribe to `$SWARM.router.sink.request-reviewer` and answer each request once, for 30 seconds; creates and writes the `notify-flows` bucket (every bucket, by its `$KV.*.>` grant)                                                                                                                                                                                                                                                                                                                                  |
-| an agent, under its own credential      | create its own consumer `todo-<agent>` in the form whose filter is `$SWARM.router.todo.<agent>`, pull from it, and ack what it delivered                                                                                                                                                                                                                                              |
+| an agent, under its own credential      | create its own consumer `todo-<agent>` in the form whose filter is `$SWARM.router.todo.<agent>`, pull from it, and ack what it delivered                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
 
-No other client may publish on `$SWARM.router.todo.*`, `$SWARM.router.sink.request-reviewer` or
+No other client may publish on `$SWARM.router.todo.*`, `$SWARM.router.log`, the two
+`$SWARM.router.api.*` subjects, `$SWARM.router.sink.request-reviewer` or
 `$SWARM.events.forge.*`, so each has one writer, and the controller alone
-hears the review requests. The review request needs no secret of its own: the
+hears the review requests and the router alone hears the api requests. The review request needs no secret of its own: the
 controller asks the forge under the token it already holds. The agent's create subject
 carries the consumer's filter, and the queue server checks it against the
 request, so an agent can't create a consumer that delivers another agent's

docs/swarm/ui.md

@@ -235,6 +235,15 @@ it read no document. The controller refuses the switch when the stored revision
 is another, including when someone wrote a document after the page read none, and
 the page shows why. Reload to switch against the current document.
 
+Beside each node in the trees, a number says how many of the router's newest 50
+forge events reached it, from a dry run of the saved flows (`GET /api/notify/dry-run`).
+Below the flows, **recent deliveries** lists what the router delivered, newest
+first: when, which flow and node, the event and issue, who got it, and whether it
+went through (`GET /api/notify/deliveries`). Both come from the router; while the
+running router is older than the controller, each shows the controller's "not
+available" message in place of its numbers or table, and the rest of the page
+works. See [deliveries and dry runs](notify-router.md#deliveries-and-dry-runs).
+
 #### Editing a flow
 
 `/notifications/<flow id>` edits one custom flow. The canvas draws the flow left

frontend/packages/swarm-ui/src/pages/notifications/Deliveries.tsx

@@ -0,0 +1,61 @@
+// <Deliveries> — the notification router's recent deliveries, newest first.
+// The router keeps the newest 500 rows; this shows what
+// `GET /api/notify/deliveries` returns.
+import { RelativeTime } from "../../ui/relative-time/RelativeTime.js";
+import { Table, type TableColumn } from "../../ui/table/Table.js";
+import { issueRef, outcomeText, type Delivery } from "./RouterLive.js";
+
+// A delivery with its place in the answer, which is what tells two identical rows
+// apart.
+type Row = Delivery & { place: number };
+
+const COLUMNS: TableColumn<Row>[] = [
+  {
+    key: "at",
+    header: "when",
+    render: (d) => <RelativeTime epochMs={d.at * 1000} />,
+    sortBy: (d) => d.at,
+  },
+  {
+    key: "flow",
+    header: "flow",
+    render: (d) => `${d.flow}/${d.node}`,
+    sortBy: (d) => d.flow,
+    filterValue: (d) => d.flow,
+  },
+  {
+    key: "event",
+    header: "event",
+    render: (d) => `${d.event} ${issueRef(d)}`,
+    filterValue: (d) => `${d.event} ${issueRef(d)}`,
+  },
+  {
+    key: "target",
+    header: "to",
+    render: (d) =>
+      d.kind === "request_reviewer" ? `review by ${d.target}` : d.target,
+    filterValue: (d) => d.target,
+  },
+  {
+    key: "outcome",
+    header: "outcome",
+    render: (d) => (
+      <span class={d.ok ? "" : "notify-failed"}>{outcomeText(d)}</span>
+    ),
+    cellClass: "ui-table-prose",
+    sortBy: (d) => (d.ok ? 0 : 1),
+  },
+];
+
+export function Deliveries({ rows }: { rows: Delivery[] }) {
+  if (rows.length === 0) return <p>no deliveries yet</p>;
+  const placed = rows.map((d, place) => ({ ...d, place }));
+  return (
+    <Table
+      columns={COLUMNS}
+      rows={placed}
+      rowKey={(d) => String(d.place)}
+      storageKey="swarm-ui:notifications:deliveries-filters"
+    />
+  );
+}

frontend/packages/swarm-ui/src/pages/notifications/NotificationsPage.css

@@ -249,3 +249,26 @@
 .notify-name {
   flex: 0 1 24em;
 }
+
+/* The dry run's counts beside each node, and the router's "not available". */
+.notify-node-hits {
+  margin-left: auto;
+  flex: none;
+  color: var(--muted);
+  font-variant-numeric: tabular-nums;
+}
+
+.notify-dry-run {
+  margin: 0;
+  color: var(--muted);
+  font-size: 0.9em;
+}
+
+.notify-unavailable {
+  margin: 0;
+  color: var(--muted);
+}
+
+.notify-failed {
+  color: var(--red);
+}

frontend/packages/swarm-ui/src/pages/notifications/NotificationsPage.tsx

@@ -19,6 +19,7 @@ import {
   type RefreshIntervalMs,
 } from "../../ui/refresh-interval/RefreshInterval.js";
 import { fetchJson, usePolled } from "../../ui/polled/usePolled.js";
+import { Deliveries } from "./Deliveries.js";
 import {
   duplicateFlow,
   newFlow,
@@ -32,11 +33,21 @@ import {
   type Flow,
   type FlowsView,
 } from "./FlowTypes.js";
+import {
+  DELIVERIES_URL,
+  DRY_RUN_URL,
+  hitsOf,
+  unavailableText,
+  type Deliveries as DeliveriesView,
+  type DryRun,
+} from "./RouterLive.js";
 import "./NotificationsPage.css";
 
 const DEFAULT_INTERVAL_MS: RefreshIntervalMs = 30_000;
 
-function FlowTree({ flow }: { flow: Flow }) {
+// `hits`, when the dry run covers this flow, puts the number of the newest events
+// that reached each node beside it.
+function FlowTree({ flow, hits }: { flow: Flow; hits?: Map<string, number> }) {
   return (
     <ul class="notify-flow-tree">
       {treeRows(flow).map(({ node, depth }) => (
@@ -47,6 +58,11 @@ function FlowTree({ flow }: { flow: Flow }) {
         >
           <span class="notify-node-kind">{node.kind}</span>
           <span class="notify-node-text">{describeNode(node)}</span>
+          {hits ? (
+            <span class="notify-node-hits" title="events of the last dry run">
+              {hits.get(node.id) ?? 0}
+            </span>
+          ) : null}
         </li>
       ))}
     </ul>
@@ -63,6 +79,18 @@ export function NotificationsPage() {
   } = usePolled(intervalMs, (signal) =>
     fetchJson<FlowsView>("/api/notify/flows", signal),
   );
+  // The router's own answers. Either is 503 while the running router predates
+  // it, which the page says in place of the section.
+  const { data: dry, error: dryError } = usePolled(intervalMs, (signal) =>
+    fetchJson<DryRun>(DRY_RUN_URL, signal),
+  );
+  const { data: deliveries, error: deliveriesError } = usePolled(
+    intervalMs,
+    (signal) => fetchJson<DeliveriesView>(DELIVERIES_URL, signal),
+  );
+  // A failed poll keeps the last answer; the counts are shown only while the
+  // latest poll worked.
+  const shown = dryError ? null : dry;
   const [pending, setPending] = useState(false);
   const [actionError, setActionError] = useState<ProblemDetails | null>(null);
   const [newName, setNewName] = useState("");
@@ -149,6 +177,15 @@ export function NotificationsPage() {
       {!error && view === null ? <p>loading…</p> : null}
       {view ? (
         <div class="notify-flows">
+          {dryError ? (
+            <p class="notify-unavailable">
+              {unavailableText("hit counts", dryError)}
+            </p>
+          ) : dry ? (
+            <p class="notify-dry-run">
+              counts: events that reached each node, of the newest {dry.events}
+            </p>
+          ) : null}
           <h3>built-in flows</h3>
           {view.builtins.map((flow) => {
             const on = !view.disabled_builtins.includes(flow.id);
@@ -164,7 +201,7 @@ export function NotificationsPage() {
                     onChange={(checked) => setBuiltin(flow.id, checked)}
                   />
                 </div>
-                <FlowTree flow={flow} />
+                <FlowTree flow={flow} hits={hitsOf(shown, flow.id)} />
                 <Button disabled={pending} onClick={() => void duplicate(flow)}>
                   duplicate to edit
                 </Button>
@@ -202,9 +239,19 @@ export function NotificationsPage() {
                   duplicate
                 </Button>
               </div>
-              <FlowTree flow={flow} />
+              <FlowTree flow={flow} hits={hitsOf(shown, flow.id)} />
             </Card>
           ))}
+          <h3>recent deliveries</h3>
+          {deliveriesError ? (
+            <p class="notify-unavailable">
+              {unavailableText("the deliveries log", deliveriesError)}
+            </p>
+          ) : deliveries ? (
+            <Deliveries rows={deliveries.rows} />
+          ) : (
+            <p>loading…</p>
+          )}
         </div>
       ) : null}
     </Panel>

frontend/packages/swarm-ui/src/pages/notifications/RouterLive.ts

@@ -0,0 +1,62 @@
+// Wire types and helpers for what the notification router reports about
+// itself: its deliveries log and a dry run of the saved flows. Backed by
+// swarm-controller's `GET /api/notify/deliveries` and
+// `GET /api/notify/dry-run`, which answer 503 while the running router
+// predates them.
+import type { ProblemDetails } from "@hive/shared/api-error.js";
+
+export const DELIVERIES_URL = "/api/notify/deliveries";
+export const DRY_RUN_URL = "/api/notify/dry-run";
+
+export interface Delivery {
+  // Unix seconds.
+  at: number;
+  flow: string;
+  node: string;
+  event: string;
+  repo: string;
+  number: number;
+  kind: "todo" | "request_reviewer";
+  target: string;
+  ok: boolean;
+  detail: string;
+}
+
+export interface Deliveries {
+  rows: Delivery[];
+}
+
+export interface DryRun {
+  // How many of the newest forge events were run.
+  events: number;
+  flows: { flow: string; nodes: { node: string; hits: number }[] }[];
+}
+
+// Hits per node id, for one flow; `undefined` when the dry run has no entry
+// for it (the flow is switched off, or there is no dry run).
+export function hitsOf(
+  dry: DryRun | null,
+  flowId: string,
+): Map<string, number> | undefined {
+  const flow = dry?.flows.find((f) => f.flow === flowId);
+  return flow && new Map(flow.nodes.map((n) => [n.node, n.hits]));
+}
+
+// `repo#n`, the way the forge names an issue or a PR.
+export function issueRef(d: Pick<Delivery, "repo" | "number">): string {
+  return `${d.repo}#${d.number}`;
+}
+
+// The outcome column: "ok" with what was done, or the error.
+export function outcomeText(d: Pick<Delivery, "ok" | "detail">): string {
+  if (!d.ok) return `error: ${d.detail}`;
+  return d.detail === "" ? "ok" : `ok: ${d.detail}`;
+}
+
+// The line shown in place of a section the router cannot serve.
+export function unavailableText(what: string, problem: ProblemDetails): string {
+  const why = problem.detail ?? problem.title ?? "no reason given";
+  return problem.status === 503
+    ? `${what} not available: ${why}`
+    : `${what} failed to load: ${why}`;
+}

frontend/packages/swarm-ui/test/router-live.test.mjs

@@ -0,0 +1,65 @@
+// The notifications page's helpers for the router's deliveries and dry run.
+// Run by `npm test`.
+import assert from "node:assert/strict";
+import { test } from "node:test";
+import {
+  hitsOf,
+  issueRef,
+  outcomeText,
+  unavailableText,
+} from "../src/pages/notifications/RouterLive.ts";
+
+const dry = {
+  events: 50,
+  flows: [
+    {
+      flow: "assigned",
+      nodes: [
+        { node: "source", hits: 49 },
+        { node: "sink", hits: 0 },
+      ],
+    },
+  ],
+};
+
+test("hitsOf maps a flow's node ids to their counts, zero included", () => {
+  const hits = hitsOf(dry, "assigned");
+  assert.equal(hits.get("source"), 49);
+  assert.equal(hits.get("sink"), 0);
+});
+
+test("hitsOf has nothing for a flow the dry run left out, or no dry run", () => {
+  assert.equal(hitsOf(dry, "closed"), undefined);
+  assert.equal(hitsOf(null, "assigned"), undefined);
+});
+
+test("a delivery reads as its issue and its outcome", () => {
+  assert.equal(issueRef({ repo: "o/r", number: 9 }), "o/r#9");
+  assert.equal(
+    outcomeText({ ok: true, detail: "upsert forge:o/r#9:assignment" }),
+    "ok: upsert forge:o/r#9:assignment",
+  );
+  assert.equal(outcomeText({ ok: true, detail: "" }), "ok");
+  assert.equal(
+    outcomeText({ ok: false, detail: "the forge said no" }),
+    "error: the forge said no",
+  );
+});
+
+test("a 503 reads as not available, any other failure as failed to load", () => {
+  assert.equal(
+    unavailableText("the deliveries log", {
+      status: 503,
+      detail: "the notification router does not answer this",
+    }),
+    "the deliveries log not available: the notification router does not answer this",
+  );
+  assert.equal(
+    unavailableText("hit counts", { status: 502, detail: "bad" }),
+    "hit counts failed to load: bad",
+  );
+  assert.equal(
+    unavailableText("hit counts", { status: 503 }),
+    "hit counts not available: no reason given",
+  );
+});

swarm-controller/README.md

@@ -85,7 +85,8 @@ The swarm's control plane. The swarm UI and `swarmctl` are its clients.
 - **Notification flows** — which of the notification router's built-in flows are
   switched off, and the custom flows, as one document in the `notify-flows`
   KV bucket, which this daemon alone writes (`GET`/`PUT /api/notify/flows`);
-  the router reads it →
+  the router reads it. `GET /api/notify/deliveries` and `GET /api/notify/dry-run` ask
+  the router for its deliveries log and a dry run of the saved flows →
   [`docs/swarm/notify-router.md`](../docs/swarm/notify-router.md#flows).
 - **Operator profiles** — each operator's name and the pronouns on their
   forge profile, for every member of authelia's operator group, as one list

swarm-controller/src/main.rs

@@ -52,6 +52,7 @@ mod matrix_account;
 mod message_stream;
 mod meta_flake;
 mod notify_flows;
+mod notify_router_api;
 mod otel_http_client;
 mod pulls;
 mod queue_identity;

swarm-controller/src/notify_flows.rs

@@ -76,7 +76,7 @@ impl FlowsView {
 }
 
 /// What the key holds.
-enum Stored {
+pub(crate) enum Stored {
     /// No key, or a deleted one.
     Nothing,
     Document(Document),
@@ -85,7 +85,7 @@ enum Stored {
 }
 
 /// The stored document, with the revision to write against.
-async fn stored(store: &Store) -> anyhow::Result<(Stored, Option<u64>)> {
+pub(crate) async fn stored(store: &Store) -> anyhow::Result<(Stored, Option<u64>)> {
     struct Read(Stored);
     impl Default for Read {
         fn default() -> Self {
@@ -103,7 +103,7 @@ async fn stored(store: &Store) -> anyhow::Result<(Stored, Option<u64>)> {
 }
 
 /// Open [`notify_flows::BUCKET`], creating it if nothing has yet.
-async fn open(client: &async_nats::Client) -> anyhow::Result<Store> {
+pub(crate) async fn open(client: &async_nats::Client) -> anyhow::Result<Store> {
     let js = async_nats::jetstream::new(client.clone());
     if let Ok(store) = js.get_key_value(notify_flows::BUCKET).await {
         return Ok(store);
@@ -119,7 +119,7 @@ async fn open(client: &async_nats::Client) -> anyhow::Result<Store> {
     .with_context(|| format!("creating the {} bucket", notify_flows::BUCKET))
 }
 
-fn queue_problem(e: &anyhow::Error) -> ProblemDetails {
+pub(crate) fn queue_problem(e: &anyhow::Error) -> ProblemDetails {
     tracing::warn!(error = %format!("{e:#}"), "notify flows: queue failure");
     error_problem(wanted_error_status(e), &format!("{e:#}"))
 }
@@ -158,7 +158,7 @@ pub(crate) async fn get_flows(
     }
 }
 
-fn unreadable(why: &DecodeError) -> ProblemDetails {
+pub(crate) fn unreadable(why: &DecodeError) -> ProblemDetails {
     error_problem(
         StatusCode::CONFLICT,
         &format!("the stored flows document is not one this build reads: {why}"),

swarm-controller/src/notify_router_api.rs

@@ -0,0 +1,335 @@
+//! `GET /api/notify/deliveries` and `GET /api/notify/dry-run`: what the
+//! notification router has delivered, and what the saved flows would do with
+//! the newest forge events.
+//!
+//! The router keeps the log and the events, so each is a NATS request to it
+//! (`swarm_wire::router_api`). A router that predates them, or none at all,
+//! has no subscriber: the queue answers "no responders" at once and these
+//! answer 503 saying so, as they do when it hears the request and stays
+//! silent. The saved document is what `GET /api/notify/flows` shows; a draft
+//! is run by sending the same request with another document.
+
+use axum::Json;
+use axum::extract::{Query, State};
+use axum::http::StatusCode;
+use problem_details::ProblemDetails;
+use serde::Deserialize;
+use serde::de::DeserializeOwned;
+use swarm_wire::notify_flows::Document;
+use swarm_wire::router_api::{
+    DELIVERIES_SUBJECT, DRY_RUN_SUBJECT, Deliveries, DeliveriesRequest, DryRun, DryRunRequest,
+    REPLY_DEADLINE, Refused,
+};
+use utoipa::IntoParams;
+
+use crate::notify_flows::{Stored, open, queue_problem, stored, unreadable};
+use crate::routes::{AppState, error_problem};
+use crate::schedules::{queue, refused};
+
+/// Ask the router on `subject` and decode its answer as `T`.
+///
+/// 503: nothing hears the request, or what hears it does not answer in time;
+/// both mean the running router does not have this feature. 502: it answered
+/// with a refusal or with something this build cannot read.
+async fn ask<T: DeserializeOwned>(
+    client: &async_nats::Client,
+    subject: &str,
+    body: Vec<u8>,
+) -> Result<T, ProblemDetails> {
+    use async_nats::RequestErrorKind;
+
+    let sent = async_nats::Request::new()
+        .payload(body.into())
+        .timeout(Some(REPLY_DEADLINE));
+    let reply = client
+        .send_request(subject.to_owned(), sent)
+        .await
+        .map_err(|e| {
+            let detail = match e.kind() {
+                RequestErrorKind::NoResponders => "the notification router does not answer this: \
+                 it is not running, or it is an older build than this controller"
+                    .to_owned(),
+                RequestErrorKind::TimedOut => format!(
+                    "the notification router sent no answer within {}s",
+                    REPLY_DEADLINE.as_secs()
+                ),
+                _ => format!("asking the notification router failed: {e}"),
+            };
+            error_problem(StatusCode::SERVICE_UNAVAILABLE, &detail)
+        })?;
+    if let Ok(answer) = serde_json::from_slice::<T>(&reply.payload) {
+        return Ok(answer);
+    }
+    let detail = match serde_json::from_slice::<Refused>(&reply.payload) {
+        Ok(Refused { error }) => format!("the notification router refused: {error}"),
+        Err(e) => format!("the notification router's answer did not decode: {e}"),
+    };
+    Err(error_problem(StatusCode::BAD_GATEWAY, &detail))
+}
+
+/// `get_deliveries`'s query.
+#[derive(Deserialize, IntoParams)]
+pub struct DeliveriesQuery {
+    /// At most this many rows; 100 when absent, and the log keeps 500.
+    limit: Option<usize>,
+}
+
+/// The newest deliveries the router made, newest first.
+#[utoipa::path(
+    get,
+    path = "/api/notify/deliveries",
+    params(DeliveriesQuery),
+    responses(
+        (status = 200, description = "the router's deliveries log, newest first: when, which \
+            flow and node, the event and issue, who got it, and whether it went through",
+            body = Deliveries),
+        (status = 502, description = "the router refused, or its answer did not decode \
+            (problem+json)", body = String),
+        (status = 503, description = "no swarm queue is configured or connected here, or the \
+            router does not answer: it is not running, or it predates the log (problem+json)",
+            body = String),
+    ),
+    tag = "notify"
+)]
+pub(crate) async fn get_deliveries(
+    State(state): State<AppState>,
+    Query(query): Query<DeliveriesQuery>,
+) -> Result<Json<Deliveries>, ProblemDetails> {
+    let client = queue(&state).map_err(refused)?;
+    let body =
+        serde_json::to_vec(&DeliveriesRequest { limit: query.limit }).expect("a request encodes");
+    ask(&client, DELIVERIES_SUBJECT, body).await.map(Json)
+}
+
+/// The saved flows run over the newest 50 forge events, delivering nothing:
+/// how many events reached each node.
+#[utoipa::path(
+    get,
+    path = "/api/notify/dry-run",
+    responses(
+        (status = 200, description = "per flow that would run, the number of the newest 50 \
+            events that reached each node", body = DryRun),
+        (status = 409, description = "the stored document is not one this build reads \
+            (problem+json)", body = String),
+        (status = 502, description = "the router refused, or its answer did not decode \
+            (problem+json)", body = String),
+        (status = 503, description = "no swarm queue is configured or connected here, or the \
+            router does not answer: it is not running, or it predates the dry run \
+            (problem+json)", body = String),
+        (status = 500, description = "the saved flows could not be read (problem+json)",
+            body = String),
+    ),
+    tag = "notify"
+)]
+pub(crate) async fn get_dry_run(
+    State(state): State<AppState>,
+) -> Result<Json<DryRun>, ProblemDetails> {
+    let client = queue(&state).map_err(refused)?;
+    let store = open(&client).await.map_err(|e| queue_problem(&e))?;
+    let (saved, _) = stored(&store).await.map_err(|e| queue_problem(&e))?;
+    let document = match saved {
+        Stored::Nothing => Document::default(),
+        Stored::Document(doc) => doc,
+        Stored::Unreadable(e) => return Err(unreadable(&e)),
+    };
+    let body = serde_json::to_vec(&DryRunRequest { document }).expect("a request encodes");
+    ask(&client, DRY_RUN_SUBJECT, body).await.map(Json)
+}
+
+#[cfg(test)]
+mod tests {
+    use futures_util::StreamExt as _;
+    use swarm_wire::notify_flows::{self, Document};
+    use swarm_wire::router_api::{FlowHits, NodeHits};
+
+    use super::*;
+    use crate::test_nats::TestNats;
+
+    fn query(limit: Option<usize>) -> Query<DeliveriesQuery> {
+        Query(DeliveriesQuery { limit })
+    }
+
+    /// Answer the next request on `subject` with `reply`, handing the request
+    /// body back.
+    async fn answer(
+        client: &async_nats::Client,
+        subject: &'static str,
+        reply: Vec<u8>,
+    ) -> tokio::task::JoinHandle<Vec<u8>> {
+        let mut requests = client.subscribe(subject).await.expect("subscribed");
+        client.flush().await.expect("flushed");
+        let client = client.clone();
+        tokio::spawn(async move {
+            let m = requests.next().await.expect("a request");
+            client
+                .publish(m.reply.expect("a reply subject"), reply.into())
+                .await
+                .expect("answered");
+            m.payload.to_vec()
+        })
+    }
+
+    /// A router that predates the log and the dry run has no subscriber: both
+    /// answer 503 with the reason, not 500 and not a hang. The control is the
+    /// same two calls with a router answering.
+    #[tokio::test]
+    async fn without_a_router_that_answers_both_are_unavailable() {
+        let nats = TestNats::start();
+        let client = nats.client().await;
+        let state = crate::agent_limits::tests::state(&client);
+
+        let deliveries = get_deliveries(State(state.clone()), query(None))
+            .await
+            .expect_err("nobody answers");
+        assert_eq!(deliveries.status, Some(StatusCode::SERVICE_UNAVAILABLE));
+        assert!(
+            deliveries
+                .detail
+                .as_deref()
+                .unwrap_or_default()
+                .contains("older build"),
+            "{deliveries:?}"
+        );
+        let dry = get_dry_run(State(state.clone()))
+            .await
+            .expect_err("nobody answers");
+        assert_eq!(dry.status, Some(StatusCode::SERVICE_UNAVAILABLE));
+        assert!(
+            dry.detail
+                .as_deref()
+                .unwrap_or_default()
+                .contains("older build"),
+            "{dry:?}"
+        );
+
+        // Control: with a router answering, the same calls are 200.
+        let rows = answer(&client, DELIVERIES_SUBJECT, br#"{"rows":[]}"#.to_vec()).await;
+        let Json(got) = get_deliveries(State(state.clone()), query(None))
+            .await
+            .expect("answered");
+        assert!(got.rows.is_empty());
+        rows.await.expect("the router task");
+        let hits = DryRun {
+            events: 0,
+            flows: Vec::new(),
+        };
+        let dry_run = answer(&client, DRY_RUN_SUBJECT, serde_json::to_vec(&hits).unwrap()).await;
+        let Json(got) = get_dry_run(State(state)).await.expect("answered");
+        assert_eq!(got, hits);
+        dry_run.await.expect("the router task");
+    }
+
+    /// No queue on this host is 503 too.
+    #[tokio::test]
+    async fn without_a_queue_both_are_unavailable() {
+        let (state, _) = crate::routes::tests::state_with_roster();
+        let a = get_deliveries(State(state.clone()), query(None))
+            .await
+            .expect_err("no queue");
+        let b = get_dry_run(State(state)).await.expect_err("no queue");
+        assert_eq!(a.status, Some(StatusCode::SERVICE_UNAVAILABLE));
+        assert_eq!(b.status, Some(StatusCode::SERVICE_UNAVAILABLE));
+    }
+
+    /// The limit reaches the router, and the dry run sends the saved document.
+    #[tokio::test]
+    async fn the_limit_and_the_saved_document_reach_the_router() {
+        let nats = TestNats::start();
+        let client = nats.client().await;
+        let state = crate::agent_limits::tests::state(&client);
+
+        let asked = answer(&client, DELIVERIES_SUBJECT, br#"{"rows":[]}"#.to_vec()).await;
+        let _ = get_deliveries(State(state.clone()), query(Some(7)))
+            .await
+            .expect("answered");
+        let request: DeliveriesRequest =
+            serde_json::from_slice(&asked.await.expect("task")).expect("a request");
+        assert_eq!(request.limit, Some(7));
+
+        // Nothing saved: the default document.
+        let reply = DryRun {
+            events: 3,
+            flows: vec![FlowHits {
+                flow: "assigned".to_owned(),
+                nodes: vec![NodeHits {
+                    node: "source".to_owned(),
+                    hits: 3,
+                }],
+            }],
+        };
+        let asked = answer(
+            &client,
+            DRY_RUN_SUBJECT,
+            serde_json::to_vec(&reply).unwrap(),
+        )
+        .await;
+        let Json(got) = get_dry_run(State(state.clone())).await.expect("answered");
+        assert_eq!(got, reply);
+        let sent: DryRunRequest =
+            serde_json::from_slice(&asked.await.expect("task")).expect("a request");
+        assert_eq!(sent.document, Document::default());
+
+        // A saved document: the router gets that one.
+        let store = open(&client).await.expect("bucket");
+        let saved = Document {
+            disabled_builtins: vec!["closed".to_owned()],
+            ..Document::default()
+        };
+        store
+            .put(
+                notify_flows::KEY,
+                serde_json::to_vec(&saved).unwrap().into(),
+            )
+            .await
+            .expect("seed");
+        let asked = answer(
+            &client,
+            DRY_RUN_SUBJECT,
+            serde_json::to_vec(&reply).unwrap(),
+        )
+        .await;
+        let _ = get_dry_run(State(state)).await.expect("answered");
+        let sent: DryRunRequest =
+            serde_json::from_slice(&asked.await.expect("task")).expect("a request");
+        assert_eq!(sent.document, saved);
+    }
+
+    /// A refusal from the router is 502 with its reason; an unreadable saved
+    /// document is 409 and never reaches the router.
+    #[tokio::test]
+    async fn a_refusal_is_502_and_an_unreadable_document_is_409() {
+        let nats = TestNats::start();
+        let client = nats.client().await;
+        let state = crate::agent_limits::tests::state(&client);
+
+        let router = answer(
+            &client,
+            DRY_RUN_SUBJECT,
+            br#"{"error":"no events"}"#.to_vec(),
+        )
+        .await;
+        let refused = get_dry_run(State(state.clone()))
+            .await
+            .expect_err("refused");
+        assert_eq!(refused.status, Some(StatusCode::BAD_GATEWAY));
+        assert!(
+            refused
+                .detail
+                .as_deref()
+                .unwrap_or_default()
+                .contains("no events")
+        );
+        router.await.expect("the router task");
+
+        let store = open(&client).await.expect("bucket");
+        store
+            .put(
+                notify_flows::KEY,
+                br#"{"version":2,"flows":[]}"#.to_vec().into(),
+            )
+            .await
+            .expect("seed");
+        let unreadable = get_dry_run(State(state)).await.expect_err("version 2");
+        assert_eq!(unreadable.status, Some(StatusCode::CONFLICT));
+    }
+}

swarm-controller/src/startup.rs

@@ -31,8 +31,9 @@ use crate::{
     agent_control, agent_icon, agent_input, agent_limits, agent_login, agent_permissions,
     agent_renewal, agent_state_stream, agent_status, agents_md, alerts, archetypes, auth, builder,
     claude_login, config_pr, forge, forge_account, github_account, inference, issue_report,
-    linked_accounts, matrix_account, message_stream, notify_flows, pulls, queue_identity,
+    linked_accounts, matrix_account, message_stream, notify_flows, notify_router_api, pulls,
-    review_classifier, schedules, status, subagent_term, term_stream, wanted, webhook,
+    queue_identity, review_classifier, schedules, status, subagent_term, term_stream, wanted,
+    webhook,
 };
 
 /// Where the daemon binds, overridable via `SWARM_CONTROLLER_SOCKET`.
@@ -643,6 +644,8 @@ pub(crate) fn build_app(state: AppState) -> axum::Router {
             review_classifier::settings::put_settings
         ))
         .routes(routes!(notify_flows::get_flows, notify_flows::put_flows))
+        .routes(routes!(notify_router_api::get_deliveries))
+        .routes(routes!(notify_router_api::get_dry_run))
         .routes(routes!(webhook::post_webhook_forge))
         .split_for_parts();
     // Just the JSON, not the UI — Swagger UI itself is nginx-hosted from

swarm-nats-auth/src/policy.rs

@@ -298,10 +298,15 @@ impl Policy {
     ///   `swarm_wire::router_sink::REQUEST_REVIEWER_SUBJECT` and hear the answer
     ///   on its own inbox. The controller is the subject's one subscriber and
     ///   holds the forge token; no wildcard over `$SWARM.router.sink`.
+    /// - **Deliveries log.** Open or create `NOTIFY_ROUTER_LOG`, publish its one
+    ///   subject, and read rows and the newest events by `STREAM.MSG.GET`.
+    /// - **Api.** Hear the two `swarm_wire::router_api` requests and answer each
+    ///   once, under `allow_responses`. The controller is their one publisher.
     fn notify_router_permissions(client_id: &str) -> Permissions {
-        use swarm_wire::{forge_events, notify_flows, notify_router, router_todo};
+        use swarm_wire::{forge_events, notify_flows, notify_router, router_api, router_todo};
 
         let events = forge_events::STREAM;
+        let log_stream = router_api::LOG_STREAM;
         let consumer = forge_events::ROUTER_CONSUMER;
         let state = format!("KV_{}", notify_router::STATE_BUCKET);
         let flows = format!("KV_{}", notify_flows::BUCKET);
@@ -326,12 +331,21 @@ impl Policy {
             format!("$JS.API.DIRECT.GET.{flows}.>"),
             format!("$JS.API.STREAM.INFO.{}", Self::agent_status_stream()),
             swarm_wire::router_sink::REQUEST_REVIEWER_SUBJECT.to_owned(),
+            format!("$JS.API.STREAM.MSG.GET.{events}"),
+            format!("$JS.API.STREAM.INFO.{log_stream}"),
+            format!("$JS.API.STREAM.CREATE.{log_stream}"),
+            format!("$JS.API.STREAM.MSG.GET.{log_stream}"),
+            router_api::LOG_SUBJECT.to_owned(),
             Self::inbox(client_id),
         ]);
         Permissions {
             publish,
-            subscribe: Some(vec![Self::inbox(client_id)]),
+            subscribe: Some(vec![
-            responses: None,
+                router_api::DELIVERIES_SUBJECT.to_owned(),
+                router_api::DRY_RUN_SUBJECT.to_owned(),
+                Self::inbox(client_id),
+            ]),
+            responses: Some(router_api::REPLY_DEADLINE),
         }
     }
 
@@ -698,6 +712,12 @@ impl Policy {
             swarm_wire::builder::LOCK_SUBJECT.to_owned(),
             // An operator's claude login step, sent as a request the same way.
             swarm_wire::claude_login::SUBJECT.to_owned(),
+            // The notification router's deliveries log and a dry run of a flows
+            // document, sent as requests: the reply comes back on this client's
+            // inbox. The router is their one subscriber; no other grant names
+            // them.
+            swarm_wire::router_api::DELIVERIES_SUBJECT.to_owned(),
+            swarm_wire::router_api::DRY_RUN_SUBJECT.to_owned(),
             // Verified forge deliveries, kept in the forge events stream for
             // the notification router. This client is their one publisher: the
             // router reads them and no hive or agent grant names them.
@@ -1748,7 +1768,9 @@ mod tests {
     #[test]
     fn the_notify_router_drains_its_own_consumer_and_writes_only_its_own_state() {
         use swarm_queue_client::client_inbox_prefix;
-        use swarm_wire::{forge_events, notify_flows, notify_router, router_sink, router_todo};
+        use swarm_wire::{
+            forge_events, notify_flows, notify_router, router_api, router_sink, router_todo,
+        };
 
         let p = with_router()
             .permissions(NOTIFY_ROUTER)
@@ -1771,6 +1793,13 @@ mod tests {
             format!("$JS.API.DIRECT.GET.KV_{state}.$KV.{state}.42.5571"),
             router_sink::REQUEST_REVIEWER_SUBJECT.to_owned(),
             "$JS.API.STREAM.INFO.KV_agent-status".to_owned(),
+            // The deliveries log: make it, append to it, read it by sequence.
+            format!("$JS.API.STREAM.CREATE.{}", router_api::LOG_STREAM),
+            format!("$JS.API.STREAM.INFO.{}", router_api::LOG_STREAM),
+            format!("$JS.API.STREAM.MSG.GET.{}", router_api::LOG_STREAM),
+            router_api::LOG_SUBJECT.to_owned(),
+            // The newest forge events, for a dry run.
+            format!("$JS.API.STREAM.MSG.GET.{events}"),
             // The flows document: read it.
             format!("$JS.API.STREAM.INFO.KV_{flows}"),
             format!(
@@ -1797,6 +1826,17 @@ mod tests {
             "$KV.agent-schedules.atlas".to_owned(),
             "$SWARM.router.sink.other".to_owned(),
             "$SWARM.router.sink.>".to_owned(),
+            // The log: no delete, no purge, no message delete; no read of the todo
+            // stream; no other subject below the router's two prefixes; and no
+            // sending of the requests it answers.
+            format!("$JS.API.STREAM.DELETE.{}", router_api::LOG_STREAM),
+            format!("$JS.API.STREAM.PURGE.{}", router_api::LOG_STREAM),
+            format!("$JS.API.STREAM.MSG.DELETE.{}", router_api::LOG_STREAM),
+            format!("$JS.API.STREAM.MSG.GET.{}", router_todo::STREAM),
+            "$SWARM.router.log.other".to_owned(),
+            "$SWARM.router.api.other".to_owned(),
+            router_api::DRY_RUN_SUBJECT.to_owned(),
+            router_api::DELIVERIES_SUBJECT.to_owned(),
             // The flows document: not write it, nor make the bucket.
             format!("$KV.{flows}.{}", notify_flows::KEY),
             format!("$JS.API.STREAM.CREATE.KV_{flows}"),
@@ -1807,8 +1847,15 @@ mod tests {
             assert!(!may(&refused), "the router may reach {refused}");
         }
         let inbox = client_inbox_prefix(NOTIFY_ROUTER);
-        assert_eq!(p.subscribe, Some(vec![format!("{inbox}.>")]));
+        assert_eq!(
-        assert_eq!(p.responses, None);
+            p.subscribe,
+            Some(vec![
+                router_api::DELIVERIES_SUBJECT.to_owned(),
+                router_api::DRY_RUN_SUBJECT.to_owned(),
+                format!("{inbox}.>"),
+            ])
+        );
+        assert_eq!(p.responses, Some(router_api::REPLY_DEADLINE));
         assert_eq!(policy().permissions(NOTIFY_ROUTER), None);
     }
 
@@ -1854,6 +1901,48 @@ mod tests {
         );
     }
 
+    /// The controller asks and the router answers: nobody else may send the
+    /// two requests, and nobody else may hear them.
+    #[test]
+    fn only_the_reader_may_ask_the_router_and_only_the_router_may_hear_it() {
+        use swarm_wire::router_api::{DELIVERIES_SUBJECT, DRY_RUN_SUBJECT, LOG_SUBJECT};
+
+        let p = with_router();
+        let grants = [
+            (
+                "the reader",
+                p.permissions("swarm-controller").expect("reader"),
+            ),
+            ("a hive", p.permissions("hive-alpha").expect("a hive")),
+            ("the router", p.permissions(NOTIFY_ROUTER).expect("router")),
+            ("the minter", p.permissions(CLAUDE_LOGIN).expect("minter")),
+            ("the builder", p.permissions(BUILDER).expect("builder")),
+            (
+                "the classifier",
+                p.permissions(CLASSIFIER).expect("classifier"),
+            ),
+            ("atlas", p.agent_token_permissions("atlas").expect("atlas")),
+        ];
+        for (who, grant) in &grants {
+            for subject in [DELIVERIES_SUBJECT, DRY_RUN_SUBJECT] {
+                let sends = grant.publish.iter().any(|g| covers(g, subject));
+                let hears = may_subscribe(grant, subject);
+                assert_eq!(sends, *who == "the reader", "{who} sends {subject}");
+                assert_eq!(hears, *who == "the router", "{who} hears {subject}");
+            }
+            // The log has one writer.
+            let appends = grant.publish.iter().any(|g| covers(g, LOG_SUBJECT));
+            assert_eq!(appends, *who == "the router", "{who} appends to the log");
+        }
+        // The router answers for as long as the controller waits.
+        let router = &grants[2].1;
+        assert!(
+            router.responses >= Some(swarm_wire::router_api::REPLY_DEADLINE),
+            "{:?}",
+            router.responses
+        );
+    }
+
     #[test]
     fn only_the_router_may_queue_an_agents_todo_and_only_the_reader_may_keep_a_forge_event() {
         use swarm_wire::{forge_events, router_todo};

swarm-notify-router/Cargo.toml

@@ -30,5 +30,9 @@ swarm-wire.workspace = true
 tokio.workspace = true
 tracing.workspace = true
 
+# A fresh directory per test `nats-server`, for its store and ports file.
+[dev-dependencies]
+tempfile = "3"
+
 [lints]
 workspace = true

swarm-notify-router/README.md

@@ -4,5 +4,8 @@ Routes forge events to agents' todo lists. It drains its consumer on the
 forge events stream that swarm-controller fills from its webhooks, runs its
 flows over each delivery, and queues the todo ops they emit on each agent's
 subject of the `ROUTER_TODOS` stream. Each agent's harness applies them.
+It keeps the newest 500 deliveries in `NOTIFY_ROUTER_LOG` and answers
+swarm-controller's requests for them and for a dry run of the flows over the
+newest 50 events.
 
 Design: [`docs/swarm/notify-router.md`](../docs/swarm/notify-router.md).

swarm-notify-router/src/api.rs

@@ -0,0 +1,288 @@
+//! The router's two answers to swarm-controller: the newest deliveries and a
+//! dry run (`swarm_wire::router_api`).
+//!
+//! Each is a NATS request. A dry run replays the newest forge events from the
+//! events stream through [`dry_run::run`] and delivers nothing, writes no
+//! snapshot and sends no review request. The router hears these on their own
+//! task, so a slow replay never holds up the events.
+
+use anyhow::Result;
+use async_nats::jetstream;
+use futures_util::StreamExt as _;
+use serde::Serialize;
+use swarm_wire::forge_events;
+use swarm_wire::router_api::{
+    DEFAULT_ROWS, DELIVERIES_SUBJECT, DRY_RUN_EVENTS, DRY_RUN_SUBJECT, Deliveries,
+    DeliveriesRequest, DryRunRequest, Refused,
+};
+
+use crate::{dry_run, log};
+
+/// Answer requests on both subjects until the connection closes.
+pub async fn serve(client: async_nats::Client) {
+    let js = jetstream::new(client.clone());
+    let (mut deliveries, mut dry) = match (
+        client.subscribe(DELIVERIES_SUBJECT).await,
+        client.subscribe(DRY_RUN_SUBJECT).await,
+    ) {
+        (Ok(a), Ok(b)) => (a, b),
+        (a, b) => {
+            let why = a
+                .err()
+                .map(|e| e.to_string())
+                .or(b.err().map(|e| e.to_string()));
+            tracing::error!(error = ?why, "notify router: cannot hear api requests; the controller will report them unavailable");
+            return;
+        }
+    };
+    tracing::info!("notify router: answering deliveries and dry-run requests");
+    loop {
+        let (request, dry_run) = tokio::select! {
+            Some(m) = deliveries.next() => (m, false),
+            Some(m) = dry.next() => (m, true),
+            else => return,
+        };
+        let (client, js) = (client.clone(), js.clone());
+        tokio::spawn(async move {
+            let Some(reply) = request.reply else { return };
+            let answer = if dry_run {
+                dry_run_answer(&js, &request.payload).await
+            } else {
+                deliveries_answer(&js, &request.payload).await
+            };
+            if let Err(e) = client.publish(reply, answer.into()).await {
+                tracing::warn!(error = %e, "notify router: could not answer an api request");
+            }
+        });
+    }
+}
+
+/// The reply body: `value`, or a [`Refused`] saying why not.
+fn body<T: Serialize>(answer: Result<T>) -> Vec<u8> {
+    match answer {
+        Ok(value) => serde_json::to_vec(&value).expect("an answer encodes"),
+        Err(e) => {
+            tracing::warn!(
+                error = format!("{e:#}"),
+                "notify router: an api request failed"
+            );
+            serde_json::to_vec(&Refused {
+                error: format!("{e:#}"),
+            })
+            .expect("a refusal encodes")
+        }
+    }
+}
+
+async fn deliveries_answer(js: &jetstream::Context, payload: &[u8]) -> Vec<u8> {
+    body(
+        async {
+            let request: DeliveriesRequest = if payload.is_empty() {
+                DeliveriesRequest::default()
+            } else {
+                serde_json::from_slice(payload)?
+            };
+            let limit = request.limit.unwrap_or(DEFAULT_ROWS);
+            Ok(Deliveries {
+                rows: log::recent(js, limit).await?,
+            })
+        }
+        .await,
+    )
+}
+
+async fn dry_run_answer(js: &jetstream::Context, payload: &[u8]) -> Vec<u8> {
+    body(
+        async {
+            let request: DryRunRequest = serde_json::from_slice(payload)?;
+            request
+                .document
+                .check()
+                .map_err(|why| anyhow::anyhow!("the document fails its check: {why}"))?;
+            let (flows, _) = request.document.active();
+            let mut events: Vec<(String, Vec<u8>)> =
+                log::newest(js, forge_events::STREAM, DRY_RUN_EVENTS)
+                    .await?
+                    .into_iter()
+                    .filter_map(|(subject, body)| {
+                        Some((forge_events::event_type(&subject)?.to_owned(), body))
+                    })
+                    .collect();
+            events.reverse();
+            Ok(dry_run::run(&flows, &events))
+        }
+        .await,
+    )
+}
+
+#[cfg(test)]
+mod tests {
+    use std::time::Duration;
+
+    use swarm_wire::notify_flows::Document;
+    use swarm_wire::router_api::{DryRun, Kind, LOG_STREAM, Row};
+    use swarm_wire::router_todo;
+
+    use super::*;
+    use crate::test_nats::TestNats;
+
+    /// Send a request to the router's api, retrying while its subscription
+    /// is not yet in place.
+    async fn ask(client: &async_nats::Client, subject: &str, body: Vec<u8>) -> Vec<u8> {
+        for _ in 0..50 {
+            if let Ok(m) = client
+                .request(subject.to_owned(), body.clone().into())
+                .await
+            {
+                return m.payload.to_vec();
+            }
+            tokio::time::sleep(Duration::from_millis(100)).await;
+        }
+        panic!("nothing answered {subject}");
+    }
+
+    fn row(n: u64) -> Row {
+        log::Cause {
+            event: "issue_assign".to_owned(),
+            repo: "o/r".to_owned(),
+            number: n,
+        }
+        .row(
+            "assigned",
+            "sink",
+            Kind::Todo,
+            "atlas",
+            Ok("upsert k".to_owned()),
+        )
+    }
+
+    async fn publish_event(js: &jetstream::Context, event: &str, n: i64) {
+        let body = serde_json::json!({
+            "action": "assigned",
+            "repository": { "full_name": "o/r", "id": 1 },
+            "issue": { "number": n, "title": "t", "html_url": "u", "assignees": [{ "login": "atlas" }] },
+            "sender": { "login": "mara" },
+        });
+        js.publish(
+            forge_events::subject(event),
+            serde_json::to_vec(&body).unwrap().into(),
+        )
+        .await
+        .expect("published")
+        .await
+        .expect("acked");
+    }
+
+    /// The newest rows come back newest first, no more than asked for, and
+    /// the stream keeps 500 of them.
+    #[tokio::test]
+    async fn the_deliveries_are_the_newest_rows_and_the_log_keeps_500() {
+        let nats = TestNats::start();
+        let client = nats.client().await;
+        let js = jetstream::new(client.clone());
+        log::open_or_create(&js).await.expect("the log stream");
+        tokio::spawn(serve(client.clone()));
+        for n in 1..=3 {
+            log::append(&js, &row(n)).await;
+        }
+
+        let all: Deliveries =
+            serde_json::from_slice(&ask(&client, DELIVERIES_SUBJECT, Vec::new()).await)
+                .expect("an answer");
+        let numbers: Vec<u64> = all.rows.iter().map(|r| r.number).collect();
+        assert_eq!(numbers, [3, 2, 1]);
+        let first = Row {
+            at: all.rows[0].at,
+            ..row(3)
+        };
+        assert_eq!(all.rows[0], first);
+
+        let two: Deliveries = serde_json::from_slice(
+            &ask(&client, DELIVERIES_SUBJECT, br#"{"limit":2}"#.to_vec()).await,
+        )
+        .expect("an answer");
+        assert_eq!(
+            two.rows.iter().map(|r| r.number).collect::<Vec<_>>(),
+            [3, 2]
+        );
+
+        for n in 4..=503 {
+            log::append(&js, &row(n)).await;
+        }
+        let mut stream = js.get_stream(LOG_STREAM).await.expect("the stream");
+        assert_eq!(stream.info().await.expect("info").state.messages, 500);
+        let newest: Deliveries = serde_json::from_slice(
+            &ask(&client, DELIVERIES_SUBJECT, br#"{"limit":9999}"#.to_vec()).await,
+        )
+        .expect("an answer");
+        assert_eq!(newest.rows.len(), 500);
+        assert_eq!(newest.rows[0].number, 503);
+        assert_eq!(newest.rows[499].number, 4);
+    }
+
+    /// A dry run counts the newest 50 events, and delivers nothing: no todo
+    /// op, no log row.
+    #[tokio::test]
+    async fn a_dry_run_counts_the_newest_fifty_events_and_delivers_nothing() {
+        let nats = TestNats::start();
+        let client = nats.client().await;
+        let js = jetstream::new(client.clone());
+        swarm_queue_client::forge_events::open_or_create(&client)
+            .await
+            .expect("events");
+        swarm_queue_client::router_todo::open_or_create(&client)
+            .await
+            .expect("todos");
+        log::open_or_create(&js).await.expect("the log stream");
+        tokio::spawn(serve(client.clone()));
+        // 60 assign events on distinct issues, then one on a type no flow
+        // starts from.
+        for n in 1..=60 {
+            publish_event(&js, "issue_assign", n).await;
+        }
+        publish_event(&js, "issue_comment", 61).await;
+
+        let request = serde_json::to_vec(&DryRunRequest {
+            document: Document::default(),
+        })
+        .unwrap();
+        let reply = ask(&client, DRY_RUN_SUBJECT, request).await;
+        let done: DryRun = serde_json::from_slice(&reply).expect("a dry run");
+        assert_eq!(done.events, 50);
+        let hits = |flow: &str, node: &str| {
+            done.flows
+                .iter()
+                .find(|f| f.flow == flow)
+                .and_then(|f| f.nodes.iter().find(|n| n.node == node))
+                .map(|n| n.hits)
+        };
+        // 49 of the newest 50 are assign events; the 50th is the comment.
+        assert_eq!(hits("assigned", "source"), Some(49));
+        assert_eq!(hits("assigned", "sink"), Some(49));
+        assert_eq!(hits("unassigned", "sink"), Some(0));
+
+        for (stream, what) in [(router_todo::STREAM, "todo op"), (LOG_STREAM, "log row")] {
+            let mut s = js.get_stream(stream).await.expect("the stream");
+            assert_eq!(s.info().await.expect("info").state.messages, 0, "a {what}");
+        }
+    }
+
+    /// A document that fails its check is refused, and says why.
+    #[tokio::test]
+    async fn a_document_that_fails_its_check_is_refused() {
+        let nats = TestNats::start();
+        let client = nats.client().await;
+        tokio::spawn(serve(client.clone()));
+        let doc = Document {
+            disabled_builtins: vec!["no-such-flow".to_owned()],
+            ..Document::default()
+        };
+        let request = serde_json::to_vec(&DryRunRequest { document: doc }).unwrap();
+        let refused: Refused =
+            serde_json::from_slice(&ask(&client, DRY_RUN_SUBJECT, request).await)
+                .expect("a refusal");
+        assert!(refused.error.contains("no-such-flow"), "{}", refused.error);
+        let garbled = ask(&client, DRY_RUN_SUBJECT, b"nonsense".to_vec()).await;
+        assert!(serde_json::from_slice::<Refused>(&garbled).is_ok());
+    }
+}

swarm-notify-router/src/dry_run.rs

@@ -0,0 +1,183 @@
+//! Running a flows document over past events without delivering anything.
+//!
+//! Each event goes through the walk the router delivers with ([`flow::reached`]
+//! and [`route::route`]), counting the nodes it reaches. The per-issue
+//! snapshots start empty and carry from one event to the next, as the router's
+//! do, so an assign event's `assignees_added` is the difference to the
+//! previous replayed one. Every recipient counts as an agent.
+
+use std::collections::BTreeMap;
+
+use swarm_wire::notify_flows::Flow;
+use swarm_wire::router_api::{DryRun, FlowHits, NodeHits};
+
+use crate::event::{Delivery, Snapshot, fields};
+use crate::{flow, route};
+
+/// `flows` over `events`, oldest first, each a forge event type and body.
+/// An event that names no issue or PR is not counted.
+pub fn run(flows: &[Flow], events: &[(String, Vec<u8>)]) -> DryRun {
+    let mut hits: Vec<BTreeMap<&str, u64>> = flows
+        .iter()
+        .map(|f| f.nodes.iter().map(|n| (n.id.as_str(), 0)).collect())
+        .collect();
+    let mut snapshots: BTreeMap<String, Snapshot> = BTreeMap::new();
+    let mut run = 0;
+    for (event_type, body) in events {
+        let Some(delivery) = Delivery::parse(body) else {
+            continue;
+        };
+        run += 1;
+        let key = delivery.state_key();
+        let before = snapshots.remove(&key).unwrap_or_default();
+        let event_fields = fields(event_type, &delivery, &before);
+        for (flow, counts) in flows.iter().zip(&mut hits) {
+            for id in flow::reached(flow, event_type, &event_fields) {
+                if let Some(count) = counts.get_mut(id) {
+                    *count += 1;
+                }
+            }
+        }
+        let routed = route::route(event_type, &delivery, &before, flows, |_| true);
+        snapshots.insert(key, routed.next);
+    }
+    DryRun {
+        events: run,
+        flows: flows
+            .iter()
+            .zip(hits)
+            .map(|(flow, counts)| FlowHits {
+                flow: flow.id.clone(),
+                nodes: flow
+                    .nodes
+                    .iter()
+                    .map(|n| NodeHits {
+                        node: n.id.clone(),
+                        hits: counts.get(n.id.as_str()).copied().unwrap_or(0),
+                    })
+                    .collect(),
+            })
+            .collect(),
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use swarm_wire::notify_flows::{Document, builtins};
+
+    use super::*;
+
+    fn body(action: &str, number: i64, assignees: &[&str]) -> Vec<u8> {
+        let people: Vec<_> = assignees
+            .iter()
+            .map(|a| serde_json::json!({ "login": a }))
+            .collect();
+        serde_json::to_vec(&serde_json::json!({
+            "action": action,
+            "repository": { "full_name": "o/r", "id": 1 },
+            "issue": { "number": number, "title": "t", "html_url": "u", "assignees": people },
+            "sender": { "login": "mara" },
+        }))
+        .expect("encodes")
+    }
+
+    fn count(r: &DryRun, flow: &str, node: &str) -> u64 {
+        r.flows
+            .iter()
+            .find(|f| f.flow == flow)
+            .and_then(|f| f.nodes.iter().find(|n| n.node == node))
+            .unwrap_or_else(|| panic!("no node {flow}/{node}"))
+            .hits
+    }
+
+    /// The built-ins over an assign, a second assign that adds nobody, an
+    /// unassign and a close: each node counts the events that reached it.
+    #[test]
+    fn each_node_counts_the_events_that_reached_it() {
+        let (flows, _) = Document::default().active();
+        let events = vec![
+            ("issue_assign".to_owned(), body("assigned", 5, &["atlas"])),
+            ("issue_assign".to_owned(), body("assigned", 5, &["atlas"])),
+            ("issue_assign".to_owned(), body("unassigned", 5, &[])),
+            ("issues".to_owned(), body("closed", 5, &[])),
+            ("issues".to_owned(), b"not json".to_vec()),
+        ];
+        let r = run(&flows, &events);
+        assert_eq!(r.events, 4);
+        // The source sees all three assign events, the match and the sink both
+        // assigned ones, though the second adds nobody.
+        assert_eq!(count(&r, "assigned", "source"), 3);
+        assert_eq!(count(&r, "assigned", "match"), 2);
+        assert_eq!(count(&r, "assigned", "sink"), 2);
+        // Only the unassign reaches the unassigned flow's sink.
+        assert_eq!(count(&r, "unassigned", "source"), 3);
+        assert_eq!(count(&r, "unassigned", "sink"), 1);
+        // The close is an `issues` event, which these flows start from only
+        // in `closed`.
+        assert_eq!(count(&r, "closed", "source"), 1);
+        assert_eq!(count(&r, "closed", "sink"), 1);
+        // No pull request push: the classifier flow counts zero.
+        assert_eq!(count(&r, "head-classifier", "source"), 0);
+        assert_eq!(count(&r, "head-classifier", "request"), 0);
+    }
+
+    /// A condition on a field that depends on the previous event of the
+    /// issue (`assignees_added`) holds for the first assign event and not for
+    /// an identical second one.
+    #[test]
+    fn the_issues_snapshot_carries_from_one_replayed_event_to_the_next() {
+        use swarm_wire::notify_flows::{Condition, Edge, Node, NodeKind};
+
+        let node = |id: &str, kind| Node {
+            id: id.to_owned(),
+            kind,
+        };
+        let edge = |from: &str, to: &str| Edge {
+            from: from.to_owned(),
+            to: to.to_owned(),
+        };
+        let flow = Flow {
+            id: "added".to_owned(),
+            name: "added".to_owned(),
+            enabled: true,
+            nodes: vec![
+                node(
+                    "source",
+                    NodeKind::Source {
+                        events: vec!["issue_assign".to_owned()],
+                    },
+                ),
+                node(
+                    "atlas",
+                    NodeKind::Match {
+                        when: vec![Condition {
+                            field: "assignees_added".to_owned(),
+                            equals: "atlas".to_owned(),
+                        }],
+                    },
+                ),
+            ],
+            edges: vec![edge("source", "atlas")],
+        };
+        let assign = ("issue_assign".to_owned(), body("assigned", 5, &["atlas"]));
+        let r = run(&[flow], &[assign.clone(), assign]);
+        assert_eq!(count(&r, "added", "source"), 2);
+        assert_eq!(count(&r, "added", "atlas"), 1);
+    }
+
+    /// Only the flows the document turns on are run.
+    #[test]
+    fn a_switched_off_built_in_is_not_listed() {
+        let doc = Document {
+            disabled_builtins: vec!["closed".to_owned()],
+            ..Document::default()
+        };
+        let (flows, _) = doc.active();
+        let r = run(&flows, &[]);
+        assert_eq!(r.events, 0);
+        let listed: Vec<&str> = r.flows.iter().map(|f| f.flow.as_str()).collect();
+        assert!(!listed.contains(&"closed"));
+        assert!(listed.contains(&"assigned"));
+        assert_eq!(listed.len(), builtins().len() - 1);
+    }
+}

swarm-notify-router/src/flow.rs

@@ -43,10 +43,10 @@ pub struct ReviewRequest {
     pub reviewer: String,
 }
 
-/// Walk `flow` for one event of type `event_type`, calling `leaf` on each
+/// Walk `flow` for one event of type `event_type`, calling `visit` on each node
-/// `todo` and `request_reviewer` node the event reaches. Assumes
+/// the event reaches: the `source`, each `match` it passes, and the `todo` and
-/// [`Flow::validate`] passed.
+/// `request_reviewer` nodes below. Assumes [`Flow::validate`] passed.
-fn walk<'f>(flow: &'f Flow, event_type: &str, fields: &Fields, mut leaf: impl FnMut(&'f Node)) {
+fn walk<'f>(flow: &'f Flow, event_type: &str, fields: &Fields, mut visit: impl FnMut(&'f Node)) {
     let Some(root) = flow
         .nodes
         .iter()
@@ -58,17 +58,30 @@ fn walk<'f>(flow: &'f Flow, event_type: &str, fields: &Fields, mut leaf: impl Fn
     while let Some(id) = stack.pop() {
         let Some(node) = flow.node(id) else { continue };
         match &node.kind {
-            NodeKind::Source { .. } => stack.extend(flow.children(id).into_iter().rev()),
+            NodeKind::Source { .. } => {
+                visit(node);
+                stack.extend(flow.children(id).into_iter().rev());
+            }
             NodeKind::Match { when } => {
                 if when.iter().all(|c| holds(c, fields)) {
+                    visit(node);
                     stack.extend(flow.children(id).into_iter().rev());
                 }
             }
-            NodeKind::Todo(_) | NodeKind::RequestReviewer { .. } => leaf(node),
+            NodeKind::Todo(_) | NodeKind::RequestReviewer { .. } => visit(node),
         }
     }
 }
 
+/// The ids of the nodes of `flow` that one event of type `event_type` reaches,
+/// a `match` node counting only when it passes. Emits nothing: this is the
+/// dry run's view of the walk that [`evaluate`] and [`requests`] make.
+pub fn reached<'f>(flow: &'f Flow, event_type: &str, fields: &Fields) -> Vec<&'f str> {
+    let mut ids = Vec::new();
+    walk(flow, event_type, fields, |node| ids.push(node.id.as_str()));
+    ids
+}
+
 /// Walk `flow` for one event of type `event_type` and return the todo ops it
 /// emits. Assumes [`Flow::validate`] passed.
 pub fn evaluate(flow: &Flow, event_type: &str, fields: &Fields) -> Vec<Action> {

swarm-notify-router/src/log.rs

@@ -0,0 +1,134 @@
+//! The deliveries log: one row on the `NOTIFY_ROUTER_LOG` stream for each
+//! delivery the router makes (`swarm_wire::router_api`), and reading the
+//! newest rows back.
+//!
+//! The stream keeps the newest 500 rows. Appending is best effort: a log that
+//! cannot be written is reported and never holds up a delivery.
+
+use std::time::{SystemTime, UNIX_EPOCH};
+
+use anyhow::{Context as _, Result};
+use async_nats::jetstream::stream::{Config, DiscardPolicy, Stream};
+use async_nats::jetstream::{self};
+use futures_util::{StreamExt as _, stream};
+use swarm_wire::router_api::{Kind, LOG_MAX_MSGS, LOG_STREAM, LOG_SUBJECT, Row};
+
+/// How many messages are fetched at once.
+const FETCH_AT_ONCE: usize = 16;
+
+/// Open the stream, creating it if nothing has yet. The router is its one
+/// writer and its one creator.
+pub async fn open_or_create(js: &jetstream::Context) -> Result<Stream> {
+    match js.get_stream(LOG_STREAM).await {
+        Ok(stream) => Ok(stream),
+        Err(e) => {
+            tracing::info!(stream = LOG_STREAM, reason = %e, "notify router: log stream not available, creating it");
+            js.create_stream(Config {
+                name: LOG_STREAM.to_owned(),
+                description: Some("Notification router deliveries, newest 500".to_owned()),
+                subjects: vec![LOG_SUBJECT.to_owned()],
+                max_messages: LOG_MAX_MSGS,
+                discard: DiscardPolicy::Old,
+                ..Default::default()
+            })
+            .await
+            .context("creating the log stream")
+        }
+    }
+}
+
+/// The event a delivery answers.
+#[derive(Clone, Debug)]
+pub struct Cause {
+    pub event: String,
+    pub repo: String,
+    pub number: u64,
+}
+
+impl Cause {
+    /// A row of `kind` made at `flow`'s `node` for `target`, which went
+    /// through when `result` is `Ok`.
+    pub fn row(
+        &self,
+        flow: &str,
+        node: &str,
+        kind: Kind,
+        target: &str,
+        result: Result<String, String>,
+    ) -> Row {
+        let (ok, detail) = match result {
+            Ok(detail) => (true, detail),
+            Err(why) => (false, why),
+        };
+        Row {
+            at: SystemTime::now()
+                .duration_since(UNIX_EPOCH)
+                .map_or(0, |d| d.as_secs()),
+            flow: flow.to_owned(),
+            node: node.to_owned(),
+            event: self.event.clone(),
+            repo: self.repo.clone(),
+            number: self.number,
+            kind,
+            target: target.to_owned(),
+            ok,
+            detail,
+        }
+    }
+}
+
+/// Append `row`; a failure is logged and dropped.
+pub async fn append(js: &jetstream::Context, row: &Row) {
+    let body = serde_json::to_vec(row).expect("a row encodes");
+    let sent = async {
+        js.publish(LOG_SUBJECT, body.into())
+            .await
+            .context("publishing")?
+            .await
+            .context("waiting for the ack")
+    };
+    if let Err(e) = sent.await {
+        tracing::warn!(error = format!("{e:#}"), flow = %row.flow, "notify router: could not log a delivery");
+    }
+}
+
+/// The payloads of the newest `limit` messages of `stream`, newest first,
+/// with their subjects. A sequence that no longer exists is skipped.
+pub async fn newest(
+    js: &jetstream::Context,
+    stream: &str,
+    limit: usize,
+) -> Result<Vec<(String, Vec<u8>)>> {
+    let stream = js
+        .get_stream(stream)
+        .await
+        .with_context(|| format!("opening {stream}"))?;
+    let state = &stream.cached_info().state;
+    let (first, last) = (state.first_sequence, state.last_sequence);
+    let count = u64::try_from(limit).unwrap_or(u64::MAX);
+    let from = first.max((last + 1).saturating_sub(count));
+    let seqs = (from..=last).rev();
+    let fetched: Vec<_> = stream::iter(seqs)
+        .map(|seq| {
+            let stream = &stream;
+            async move { stream.get_raw_message(seq).await }
+        })
+        .buffered(FETCH_AT_ONCE)
+        .collect()
+        .await;
+    Ok(fetched
+        .into_iter()
+        .filter_map(Result::ok)
+        .map(|m| (m.subject.to_string(), m.payload.to_vec()))
+        .collect())
+}
+
+/// The newest `limit` rows of the log, newest first. A row that does not
+/// decode is skipped.
+pub async fn recent(js: &jetstream::Context, limit: usize) -> Result<Vec<Row>> {
+    Ok(newest(js, LOG_STREAM, limit)
+        .await?
+        .into_iter()
+        .filter_map(|(_, body)| serde_json::from_slice(&body).ok())
+        .collect())
+}

swarm-notify-router/src/main.rs

@@ -9,10 +9,15 @@
 //!
 //! A `request_reviewer` node is a request to swarm-controller, which holds the
 //! forge token (`swarm_wire::router_sink`). It is sent once, after the event's
-//! ops are queued and its snapshot is written, and the answer is only logged:
+//! ops are queued and its snapshot is written, and the answer is only logged and
+//! recorded:
 //! a controller that is down or slow must not hold up every agent's todos, and
 //! the next push asks again.
 //!
+//! Each delivery, made or failed, is a row of the deliveries log ([`log`]), and
+//! the router answers swarm-controller's requests for the newest rows and for a
+//! dry run of a flows document ([`api`], [`dry_run`]).
+//!
 //! Per issue it keeps a snapshot in its state bucket ([`event::Snapshot`]):
 //! the assignees it last saw and the agents it queued a todo for. An event is
 //! acked once its ops are queued and the snapshot is written, so a crash
@@ -32,14 +37,20 @@ use async_nats::jetstream::{self, consumer::pull, kv, message::PublishMessage};
 use futures_util::{StreamExt as _, TryStreamExt as _};
 use swarm_queue_client::{ClientSecret, QueueConfig};
 use swarm_secret_client::stored_secret::StoredSecret;
+use swarm_wire::router_api::Kind;
 use swarm_wire::router_sink::{self, Outcome, RequestReviewer};
 use swarm_wire::{forge_events, notify_router, router_todo};
 
+mod api;
+mod dry_run;
 mod event;
 mod flow;
+mod log;
 mod route;
 mod settle;
 mod stored;
+#[cfg(test)]
+mod test_nats;
 
 /// The cert-auth role this service logs in to the store as. `swarm-bao.nix`
 /// writes it, matching the leaf's subject.
@@ -93,6 +104,7 @@ async fn main() -> Result<()> {
     .await?;
 
     let mut router = Router::open(&queue).await?;
+    tokio::spawn(api::serve(queue.clone()));
     tracing::info!(
         stream = forge_events::STREAM,
         "notify router: routing forge events"
@@ -126,6 +138,9 @@ impl Router {
         swarm_queue_client::router_todo::open_or_create(queue)
             .await
             .context("opening the router todo stream")?;
+        log::open_or_create(&js)
+            .await
+            .context("opening the deliveries log stream")?;
         let events = events
             .create_consumer(pull::Config {
                 durable_name: Some(forge_events::ROUTER_CONSUMER.to_owned()),
@@ -199,8 +214,13 @@ impl Router {
         let routed = route::route(&event_type, &delivery, &snapshot, &self.flows.active, |a| {
             roster.contains(a)
         });
+        let cause = log::Cause {
+            event: event_type.clone(),
+            repo: delivery.repo.clone(),
+            number: u64::try_from(delivery.number).unwrap_or_default(),
+        };
         for action in &routed.actions {
-            self.queue(seq, action).await.map_err(Skip::Retry)?;
+            self.queue(seq, &cause, action).await.map_err(Skip::Retry)?;
         }
         if !routed.skipped.is_empty() {
             tracing::info!(%event_type, repo = %delivery.repo, number = delivery.number, skipped = ?routed.skipped, "notify router: recipients that are not agents got nothing");
@@ -210,14 +230,14 @@ impl Router {
             self.state.put(&key, bytes.into()).await.map_err(retry)?;
         }
         for request in &routed.requests {
-            self.request_review(request).await;
+            self.request_review(&cause, request).await;
         }
         Ok(())
     }
 
-    /// Ask swarm-controller for one review and log what it answered. One
+    /// Ask swarm-controller for one review, log what it answered and record
-    /// attempt: nothing here fails the event.
+    /// the delivery. One attempt: nothing here fails the event.
-    async fn request_review(&self, request: &flow::ReviewRequest) {
+    async fn request_review(&self, cause: &log::Cause, request: &flow::ReviewRequest) {
         let wire = RequestReviewer {
             repo: request.repo.clone(),
             number: request.number,
@@ -233,7 +253,7 @@ impl Router {
             request.number,
             &request.reviewer,
         );
-        match self
+        let result = match self
             .nats
             .send_request(router_sink::REQUEST_REVIEWER_SUBJECT, sent)
             .await
@@ -241,22 +261,46 @@ impl Router {
             Ok(reply) => match Outcome::decode(&String::from_utf8_lossy(&reply.payload)) {
                 Outcome::Requested => {
                     tracing::info!(%flow, %repo, number, %reviewer, "notify router: review requested");
+                    Ok("requested".to_owned())
                 }
                 Outcome::Skipped(why) => {
                     tracing::info!(%flow, %repo, number, %reviewer, %why, "notify router: review not requested");
+                    Ok(format!("skipped: {why}"))
                 }
                 Outcome::Error(why) => {
                     tracing::warn!(%flow, %repo, number, %reviewer, %why, "notify router: the controller could not request the review");
+                    Err(why)
                 }
             },
             Err(e) => {
                 tracing::warn!(%flow, %repo, number, %reviewer, error = %e, "notify router: no answer to a review request; not retried");
+                Err(format!("no answer: {e}"))
             }
-        }
+        };
+        let row = cause.row(flow, &request.node, Kind::RequestReviewer, reviewer, result);
+        log::append(&self.js, &row).await;
     }
 
-    /// Queue one op on its agent's subject and wait for the stream's ack.
+    /// Queue one op on its agent's subject, wait for the stream's ack and
-    async fn queue(&self, seq: u64, action: &flow::Action) -> Result<()> {
+    /// record the delivery, failed or not.
+    async fn queue(&self, seq: u64, cause: &log::Cause, action: &flow::Action) -> Result<()> {
+        let sent = self.publish(seq, action).await;
+        let result = match &sent {
+            Ok(()) => Ok(format!("{} {}", op_name(action.op.op), action.op.key)),
+            Err(e) => Err(format!("{e:#}")),
+        };
+        let row = cause.row(
+            &action.flow,
+            &action.node,
+            Kind::Todo,
+            &action.agent,
+            result,
+        );
+        log::append(&self.js, &row).await;
+        sent
+    }
+
+    async fn publish(&self, seq: u64, action: &flow::Action) -> Result<()> {
         let subject = router_todo::subject(&action.agent);
         let body = serde_json::to_vec(&action.op).expect("an op encodes");
         let id = format!("{seq}:{}:{}:{}", action.flow, action.node, action.agent);
@@ -319,6 +363,14 @@ fn retry(e: impl std::error::Error + Send + Sync + 'static) -> Skip {
     Skip::Retry(e.into())
 }
 
+/// The word the log uses for an op.
+const fn op_name(op: router_todo::Op) -> &'static str {
+    match op {
+        router_todo::Op::Upsert => "upsert",
+        router_todo::Op::Clear => "clear",
+    }
+}
+
 async fn client_secret(client_id: &str) -> Result<String> {
     use swarm_secret_client::builder::{ClientSecret, client_secret_path};
 
@@ -338,3 +390,148 @@ async fn client_secret(client_id: &str) -> Result<String> {
         ),
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use swarm_wire::notify_flows::builtins;
+    use swarm_wire::router_api::Row;
+
+    use super::*;
+    use crate::test_nats::TestNats;
+
+    fn cause() -> log::Cause {
+        log::Cause {
+            event: "issue_assign".to_owned(),
+            repo: "o/r".to_owned(),
+            number: 9,
+        }
+    }
+
+    fn action() -> flow::Action {
+        let flow = builtins().remove(0);
+        let (agent, op) = (
+            "atlas".to_owned(),
+            router_todo::RouterTodoOp {
+                op: router_todo::Op::Upsert,
+                key: "forge:o/r#9:assignment".to_owned(),
+                prefix: false,
+                summary: "s".to_owned(),
+                source: "router:assigned".to_owned(),
+            },
+        );
+        flow::Action {
+            flow: flow.id,
+            node: "sink".to_owned(),
+            agent,
+            op,
+        }
+    }
+
+    fn review() -> flow::ReviewRequest {
+        flow::ReviewRequest {
+            flow: "head-classifier".to_owned(),
+            node: "request".to_owned(),
+            repo: "o/r".to_owned(),
+            number: 9,
+            reviewer: "review-classifier".to_owned(),
+        }
+    }
+
+    async fn rows(router: &Router) -> Vec<Row> {
+        log::recent(&router.js, 10).await.expect("the log")
+    }
+
+    /// A todo op that was queued is a row that went through; one the stream
+    /// would not take is a row with the error, and the delivery still fails.
+    #[tokio::test]
+    async fn a_queued_todo_is_a_row_and_a_failed_one_is_a_row_with_the_error() {
+        let nats = TestNats::start();
+        let client = nats.client().await;
+        let router = Router::open(&client).await.expect("opens");
+
+        router.queue(5, &cause(), &action()).await.expect("queued");
+        let [row] = &rows(&router).await[..] else {
+            panic!("one row")
+        };
+        assert!(row.ok, "{row:?}");
+        assert_eq!(
+            (row.flow.as_str(), row.node.as_str(), row.target.as_str()),
+            ("assigned", "sink", "atlas")
+        );
+        assert_eq!(
+            (row.event.as_str(), row.repo.as_str(), row.number),
+            ("issue_assign", "o/r", 9)
+        );
+        assert_eq!(row.kind, Kind::Todo);
+        assert_eq!(row.detail, "upsert forge:o/r#9:assignment");
+
+        router
+            .js
+            .delete_stream(router_todo::STREAM)
+            .await
+            .expect("deleted");
+        router
+            .queue(6, &cause(), &action())
+            .await
+            .expect_err("no stream takes it");
+        let rows = rows(&router).await;
+        assert_eq!(rows.len(), 2);
+        assert!(!rows[0].ok && !rows[0].detail.is_empty(), "{:?}", rows[0]);
+    }
+
+    /// Each answer the controller can give is a row: requested and skipped
+    /// went through, an error did not, and no answer at all did not either.
+    #[tokio::test]
+    async fn a_review_request_is_a_row_whatever_the_controller_answers() {
+        let nats = TestNats::start();
+        let client = nats.client().await;
+        let router = Router::open(&client).await.expect("opens");
+
+        // Nobody answers.
+        router.request_review(&cause(), &review()).await;
+        let first = rows(&router).await.remove(0);
+        assert!(
+            !first.ok && first.detail.starts_with("no answer"),
+            "{first:?}"
+        );
+        assert_eq!(first.kind, Kind::RequestReviewer);
+        assert_eq!(first.target, "review-classifier");
+
+        let mut hears = client
+            .subscribe(router_sink::REQUEST_REVIEWER_SUBJECT)
+            .await
+            .expect("subscribed");
+        let answering = client.clone();
+        tokio::spawn(async move {
+            for line in [
+                "requested",
+                "skipped:a review stands",
+                "error:the forge said no",
+            ] {
+                let m = hears.next().await.expect("a request");
+                answering
+                    .publish(m.reply.expect("a reply subject"), line.into())
+                    .await
+                    .expect("answered");
+            }
+        });
+        for _ in 0..3 {
+            router.request_review(&cause(), &review()).await;
+        }
+        let got: Vec<(bool, String)> = rows(&router)
+            .await
+            .into_iter()
+            .take(3)
+            .rev()
+            .map(|r| (r.ok, r.detail))
+            .collect();
+        assert_eq!(
+            got,
+            [
+                (true, "requested".to_owned()),
+                (true, "skipped: a review stands".to_owned()),
+                (false, "the forge said no".to_owned()),
+            ]
+        );
+    }
+}

swarm-notify-router/src/test_nats.rs

@@ -0,0 +1,77 @@
+//! A throwaway `nats-server` with `JetStream`, for the tests whose subject is
+//! the server's own behaviour: what a stream keeps and what a request gets.
+//!
+//! The same shape as `swarm-queue-client`'s: the binary comes from
+//! [`NATS_SERVER_ENV`], which `nix/checks.nix` and `nix/devshell.nix` export,
+//! and a test that needs it panics when it is unset rather than skipping.
+
+use std::path::Path;
+use std::process::{Child, Command, Stdio};
+use std::time::Duration;
+
+/// The variable naming the `nats-server` binary.
+const NATS_SERVER_ENV: &str = "HIVE_NATS_SERVER";
+
+/// The subdirectory holding nothing but the server's ports file.
+const PORTS_DIR: &str = "ports";
+
+/// A running server on a loopback port it bound itself. Dropping it kills
+/// the server and removes its directory.
+pub struct TestNats {
+    child: Child,
+    dir: tempfile::TempDir,
+}
+
+impl TestNats {
+    pub fn start() -> Self {
+        let bin = std::env::var(NATS_SERVER_ENV).unwrap_or_else(|_| {
+            panic!("{NATS_SERVER_ENV} is exported by nix/checks.nix and nix/devshell.nix")
+        });
+        let dir = tempfile::Builder::new()
+            .prefix("test-nats-")
+            .tempdir()
+            .expect("a fresh directory for the server");
+        std::fs::create_dir(dir.path().join(PORTS_DIR)).expect("the ports file directory");
+        let child = Command::new(&bin)
+            .args(["-a", "127.0.0.1", "-p", "-1", "-js", "-sd"])
+            .arg(dir.path().join("store"))
+            .arg("--ports_file_dir")
+            .arg(dir.path().join(PORTS_DIR))
+            .stdout(Stdio::null())
+            .stderr(Stdio::null())
+            .spawn()
+            .unwrap_or_else(|e| panic!("starting {bin}: {e}"));
+        Self { child, dir }
+    }
+
+    /// A client connected to the server, once it accepts connections.
+    pub async fn client(&self) -> async_nats::Client {
+        for _ in 0..100 {
+            if let Some(url) = announced_url(&self.dir.path().join(PORTS_DIR))
+                && let Ok(client) = async_nats::connect(&url).await
+            {
+                return client;
+            }
+            tokio::time::sleep(Duration::from_millis(100)).await;
+        }
+        panic!(
+            "nats-server in {} never accepted a connection",
+            self.dir.path().display()
+        );
+    }
+}
+
+/// The client URL in the ports file the server writes once it listens.
+fn announced_url(dir: &Path) -> Option<String> {
+    let ports = std::fs::read_dir(dir).ok()?.next()?.ok()?;
+    let ports: serde_json::Value =
+        serde_json::from_slice(&std::fs::read(ports.path()).ok()?).ok()?;
+    ports["nats"][0].as_str().map(str::to_owned)
+}
+
+impl Drop for TestNats {
+    fn drop(&mut self) {
+        let _ = self.child.kill();
+        let _ = self.child.wait();
+    }
+}

swarm-wire/src/lib.rs

@@ -21,6 +21,7 @@ pub mod notices;
 pub mod notify_flows;
 pub mod notify_router;
 pub mod review_classifier;
+pub mod router_api;
 pub mod router_sink;
 pub mod router_todo;
 pub mod subagent_term;

swarm-wire/src/router_api.rs

@@ -0,0 +1,160 @@
+//! What the notification router records and answers, for swarm-controller's
+//! `GET /api/notify/deliveries` and `GET /api/notify/dry-run`.
+//!
+//! The router appends one [`Row`] to the [`LOG_STREAM`] stream for each
+//! delivery it makes, and keeps the newest [`LOG_MAX_MSGS`]. The controller
+//! asks it two things as NATS requests: the newest rows ([`DELIVERIES_SUBJECT`])
+//! and, for a flows document, how many of the newest forge events reach each
+//! node ([`DRY_RUN_SUBJECT`]), run without delivering anything.
+//!
+//! Three crates name these: the router records and answers, the controller
+//! asks, and the auth-callout responder grants each its side. A router that
+//! predates this answers neither, so the controller reports that.
+
+use std::time::Duration;
+
+use serde::{Deserialize, Serialize};
+
+use crate::notify_flows::Document;
+
+/// The stream holding the deliveries log.
+pub const LOG_STREAM: &str = "NOTIFY_ROUTER_LOG";
+
+/// The one subject [`LOG_STREAM`] captures. The router is its one publisher.
+pub const LOG_SUBJECT: &str = "$SWARM.router.log";
+
+/// How many rows the stream keeps; the oldest goes first.
+pub const LOG_MAX_MSGS: i64 = 500;
+
+/// The request for the newest [`Row`]s ([`DeliveriesRequest`]).
+pub const DELIVERIES_SUBJECT: &str = "$SWARM.router.api.deliveries";
+
+/// The request to run a document over the newest forge events
+/// ([`DryRunRequest`]).
+pub const DRY_RUN_SUBJECT: &str = "$SWARM.router.api.dry-run";
+
+/// How many of the newest forge events a dry run covers.
+pub const DRY_RUN_EVENTS: usize = 50;
+
+/// The rows a request gets when it names no limit.
+pub const DEFAULT_ROWS: usize = 100;
+
+/// How long the controller waits for the router. The router's reply grant
+/// lasts this long.
+pub const REPLY_DEADLINE: Duration = Duration::from_secs(10);
+
+/// What a delivery was.
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
+#[cfg_attr(feature = "schema", derive(utoipa::ToSchema))]
+#[serde(rename_all = "snake_case")]
+pub enum Kind {
+    /// A todo op queued for an agent.
+    Todo,
+    /// A request to the controller for a review.
+    RequestReviewer,
+}
+
+/// One delivery the router made.
+#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
+#[cfg_attr(feature = "schema", derive(utoipa::ToSchema))]
+pub struct Row {
+    /// Unix seconds.
+    pub at: u64,
+    pub flow: String,
+    pub node: String,
+    /// The forge event type, such as `issue_assign`.
+    pub event: String,
+    /// `owner/name` of the repository the event concerns.
+    pub repo: String,
+    /// The issue or pull request number.
+    pub number: u64,
+    pub kind: Kind,
+    /// The agent that got the todo, or the reviewer asked for.
+    pub target: String,
+    /// Whether it went through. A review the controller skipped went through.
+    pub ok: bool,
+    /// What happened: the op and key, the controller's answer, or the error.
+    pub detail: String,
+}
+
+/// Ask for the newest rows, newest first.
+#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
+pub struct DeliveriesRequest {
+    /// At most this many; [`DEFAULT_ROWS`] when absent. The log keeps
+    /// [`LOG_MAX_MSGS`], so no answer is longer.
+    #[serde(default)]
+    pub limit: Option<usize>,
+}
+
+/// The router's answer to a [`DeliveriesRequest`].
+#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
+#[cfg_attr(feature = "schema", derive(utoipa::ToSchema))]
+pub struct Deliveries {
+    /// Newest first.
+    pub rows: Vec<Row>,
+}
+
+/// Run `document` over the newest [`DRY_RUN_EVENTS`] forge events.
+#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
+pub struct DryRunRequest {
+    pub document: Document,
+}
+
+/// How many events reached one node.
+#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
+#[cfg_attr(feature = "schema", derive(utoipa::ToSchema))]
+pub struct NodeHits {
+    pub node: String,
+    /// Events that reached the node. A `match` node counts those that passed
+    /// it, a `source` node those it starts from.
+    pub hits: u64,
+}
+
+/// The hits of one flow that would run.
+#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
+#[cfg_attr(feature = "schema", derive(utoipa::ToSchema))]
+pub struct FlowHits {
+    pub flow: String,
+    /// Every node of the flow, those no event reached with `0`.
+    pub nodes: Vec<NodeHits>,
+}
+
+/// The router's answer to a [`DryRunRequest`] it ran.
+#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
+#[cfg_attr(feature = "schema", derive(utoipa::ToSchema))]
+pub struct DryRun {
+    /// How many events were run, at most [`DRY_RUN_EVENTS`]: fewer when the
+    /// stream holds fewer.
+    pub events: u64,
+    /// The flows the document turns on: the built-ins it leaves on and its
+    /// custom flows that are enabled.
+    pub flows: Vec<FlowHits>,
+}
+
+/// The router's answer to a request it could not serve.
+#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
+pub struct Refused {
+    pub error: String,
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    /// The log keeps the rows the deliveries view can ask for.
+    #[test]
+    fn a_request_never_asks_for_more_than_the_stream_keeps() {
+        assert!(i64::try_from(DEFAULT_ROWS).expect("small") <= LOG_MAX_MSGS);
+    }
+
+    #[test]
+    fn a_request_without_a_limit_decodes() {
+        let r: DeliveriesRequest = serde_json::from_str("{}").expect("empty request");
+        assert_eq!(r.limit, None);
+    }
+
+    #[test]
+    fn the_reply_deadline_leaves_the_controller_time_to_answer() {
+        assert!(REPLY_DEADLINE < Duration::from_secs(30));
+    }
+}