notify router: deliveries log, dry-run hit counts, api and page
This commit is contained in:
atlas
parent
6dd9d0086e
commit
ed1fd053e7
24 changed files with 1848 additions and 40 deletions
1
Cargo.lock
generated
1
Cargo.lock
generated
|
|
@ -5054,6 +5054,7 @@ dependencies = [
|
||||||
"swarm-queue-client",
|
"swarm-queue-client",
|
||||||
"swarm-secret-client",
|
"swarm-secret-client",
|
||||||
"swarm-wire",
|
"swarm-wire",
|
||||||
|
"tempfile",
|
||||||
"tokio",
|
"tokio",
|
||||||
"tracing",
|
"tracing",
|
||||||
]
|
]
|
||||||
|
|
|
||||||
|
|
@ -171,17 +171,68 @@ controller that predates it, gets no answer to the request and logs that. A cont
|
||||||
router runs the built-ins as before. A router that predates this ignores the
|
router runs the built-ins as before. A router that predates this ignores the
|
||||||
document, so the page and the API show a state the router isn't yet running.
|
document, so the page and the API show a state the router isn't yet running.
|
||||||
|
|
||||||
|
## Deliveries and dry runs
|
||||||
|
|
||||||
|
**The deliveries log.** The router appends one row to the `NOTIFY_ROUTER_LOG`
|
||||||
|
stream for each delivery it makes, and the stream keeps the newest 500. A
|
||||||
|
delivery is a todo op queued for an agent or a review requested from the
|
||||||
|
controller, whether it went through or not. A row has:
|
||||||
|
|
||||||
|
| Field | Holds |
|
||||||
|
| ------------------------- | ---------------------------------------------------------------------------------- |
|
||||||
|
| `at` | Unix seconds |
|
||||||
|
| `flow`, `node` | the flow and the node that delivered |
|
||||||
|
| `event`, `repo`, `number` | the forge event type and the issue or PR it concerns |
|
||||||
|
| `kind`, `target` | `todo` and the agent, or `request_reviewer` and the reviewer |
|
||||||
|
| `ok`, `detail` | whether it went through, and the op and key, the controller's answer, or the error |
|
||||||
|
|
||||||
|
A review the controller skipped (`skipped: <why>`) is `ok`: nothing was
|
||||||
|
wrong. An event the router retries after a failed step writes its rows again,
|
||||||
|
so a retry shows twice. When the router fails to write a row, it logs the failure and the delivery
|
||||||
|
stands.
|
||||||
|
|
||||||
|
**The dry run.** The router runs a flows document over the newest 50 events of
|
||||||
|
the `SWARM_EVENTS` stream and returns, for each flow the document turns on,
|
||||||
|
how many of those events reached each node. A `match` node counts the events
|
||||||
|
that passed it. It uses the walk the router delivers with and delivers nothing:
|
||||||
|
no todo op, no review request, no log row and no snapshot written. It keeps
|
||||||
|
its own per-issue snapshots, starting empty, so the first assign event it
|
||||||
|
replays for an issue counts every assignee as newly added. Every recipient
|
||||||
|
counts as an agent.
|
||||||
|
|
||||||
|
**The API.** The router answers two NATS requests from swarm-controller, which
|
||||||
|
serves them next to `GET /api/notify/flows`:
|
||||||
|
|
||||||
|
| Route | Router subject | Answers |
|
||||||
|
| ---------------------------- | ------------------------------ | ------------------------------------------------------------------ |
|
||||||
|
| `GET /api/notify/deliveries` | `$SWARM.router.api.deliveries` | the newest rows, newest first; `?limit=` up to 500, 100 by default |
|
||||||
|
| `GET /api/notify/dry-run` | `$SWARM.router.api.dry-run` | the saved flows document, run over the newest 50 events |
|
||||||
|
|
||||||
|
The dry-run request carries a whole document, so a draft runs the same way.
|
||||||
|
The controller sends the saved one. [`/notifications`](ui.md#notification-flows)
|
||||||
|
shows both: the recent deliveries, and the counts beside each node.
|
||||||
|
|
||||||
|
**Deploying in any order.** A router that predates these subjects doesn't
|
||||||
|
subscribe to them, and the queue answers the controller's request at once with
|
||||||
|
no responders. Both routes then answer 503 saying the router doesn't answer
|
||||||
|
and may be an older build, and the page shows that in place of the table and
|
||||||
|
the counts. A request the router hears and doesn't answer within 10 seconds is
|
||||||
|
the same 503. A router that answers with a refusal, or with something the
|
||||||
|
controller can't read, is 502. A controller that predates the routes has no
|
||||||
|
route: the page, if newer, shows the failure.
|
||||||
|
|
||||||
## Credentials and grants
|
## Credentials and grants
|
||||||
|
|
||||||
| Principal | Grant |
|
| Principal | Grant |
|
||||||
| --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
| --------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||||
| `swarm-notify-router`, in its container | store: `read` on `swarm/services/<router client id>/oidc/client`. Queue: its own consumer on `SWARM_EVENTS` (create, pull, ack), publish on `$SWARM.router.todo.*` and on `$SWARM.router.sink.request-reviewer`, create `ROUTER_TODOS`, its own `notify-router-state` bucket, read on the `notify-flows` bucket, and `STREAM.INFO` on the agent-status bucket, which lists the agents |
|
| `swarm-notify-router`, in its container | store: `read` on `swarm/services/<router client id>/oidc/client`. Queue: its own consumer on `SWARM_EVENTS` (create, pull, ack) and `STREAM.MSG.GET` on it for the dry run, publish on `$SWARM.router.todo.*` and on `$SWARM.router.sink.request-reviewer`, create `ROUTER_TODOS`, create `NOTIFY_ROUTER_LOG`, publish its one subject and read it by sequence, its own `notify-router-state` bucket, read on the `notify-flows` bucket, `STREAM.INFO` on the agent-status bucket, which lists the agents, and subscribe to the two `$SWARM.router.api.*` request subjects and answer each once, for 10 seconds |
|
||||||
| swarm-controller | publish on `$SWARM.events.forge.*`; subscribe to `$SWARM.router.sink.request-reviewer` and answer each request once, for 30 seconds; creates and writes the `notify-flows` bucket (every bucket, by its `$KV.*.>` grant) |
|
| swarm-controller | publish on `$SWARM.events.forge.*` and on the two `$SWARM.router.api.*` request subjects; subscribe to `$SWARM.router.sink.request-reviewer` and answer each request once, for 30 seconds; creates and writes the `notify-flows` bucket (every bucket, by its `$KV.*.>` grant) |
|
||||||
| an agent, under its own credential | create its own consumer `todo-<agent>` in the form whose filter is `$SWARM.router.todo.<agent>`, pull from it, and ack what it delivered |
|
| an agent, under its own credential | create its own consumer `todo-<agent>` in the form whose filter is `$SWARM.router.todo.<agent>`, pull from it, and ack what it delivered |
|
||||||
|
|
||||||
No other client may publish on `$SWARM.router.todo.*`, `$SWARM.router.sink.request-reviewer` or
|
No other client may publish on `$SWARM.router.todo.*`, `$SWARM.router.log`, the two
|
||||||
|
`$SWARM.router.api.*` subjects, `$SWARM.router.sink.request-reviewer` or
|
||||||
`$SWARM.events.forge.*`, so each has one writer, and the controller alone
|
`$SWARM.events.forge.*`, so each has one writer, and the controller alone
|
||||||
hears the review requests. The review request needs no secret of its own: the
|
hears the review requests and the router alone hears the api requests. The review request needs no secret of its own: the
|
||||||
controller asks the forge under the token it already holds. The agent's create subject
|
controller asks the forge under the token it already holds. The agent's create subject
|
||||||
carries the consumer's filter, and the queue server checks it against the
|
carries the consumer's filter, and the queue server checks it against the
|
||||||
request, so an agent can't create a consumer that delivers another agent's
|
request, so an agent can't create a consumer that delivers another agent's
|
||||||
|
|
|
||||||
|
|
@ -235,6 +235,15 @@ it read no document. The controller refuses the switch when the stored revision
|
||||||
is another, including when someone wrote a document after the page read none, and
|
is another, including when someone wrote a document after the page read none, and
|
||||||
the page shows why. Reload to switch against the current document.
|
the page shows why. Reload to switch against the current document.
|
||||||
|
|
||||||
|
Beside each node in the trees, a number says how many of the router's newest 50
|
||||||
|
forge events reached it, from a dry run of the saved flows (`GET /api/notify/dry-run`).
|
||||||
|
Below the flows, **recent deliveries** lists what the router delivered, newest
|
||||||
|
first: when, which flow and node, the event and issue, who got it, and whether it
|
||||||
|
went through (`GET /api/notify/deliveries`). Both come from the router; while the
|
||||||
|
running router is older than the controller, each shows the controller's "not
|
||||||
|
available" message in place of its numbers or table, and the rest of the page
|
||||||
|
works. See [deliveries and dry runs](notify-router.md#deliveries-and-dry-runs).
|
||||||
|
|
||||||
#### Editing a flow
|
#### Editing a flow
|
||||||
|
|
||||||
`/notifications/<flow id>` edits one custom flow. The canvas draws the flow left
|
`/notifications/<flow id>` edits one custom flow. The canvas draws the flow left
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,61 @@
|
||||||
|
// <Deliveries> — the notification router's recent deliveries, newest first.
|
||||||
|
// The router keeps the newest 500 rows; this shows what
|
||||||
|
// `GET /api/notify/deliveries` returns.
|
||||||
|
import { RelativeTime } from "../../ui/relative-time/RelativeTime.js";
|
||||||
|
import { Table, type TableColumn } from "../../ui/table/Table.js";
|
||||||
|
import { issueRef, outcomeText, type Delivery } from "./RouterLive.js";
|
||||||
|
|
||||||
|
// A delivery with its place in the answer, which is what tells two identical rows
|
||||||
|
// apart.
|
||||||
|
type Row = Delivery & { place: number };
|
||||||
|
|
||||||
|
const COLUMNS: TableColumn<Row>[] = [
|
||||||
|
{
|
||||||
|
key: "at",
|
||||||
|
header: "when",
|
||||||
|
render: (d) => <RelativeTime epochMs={d.at * 1000} />,
|
||||||
|
sortBy: (d) => d.at,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
key: "flow",
|
||||||
|
header: "flow",
|
||||||
|
render: (d) => `${d.flow}/${d.node}`,
|
||||||
|
sortBy: (d) => d.flow,
|
||||||
|
filterValue: (d) => d.flow,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
key: "event",
|
||||||
|
header: "event",
|
||||||
|
render: (d) => `${d.event} ${issueRef(d)}`,
|
||||||
|
filterValue: (d) => `${d.event} ${issueRef(d)}`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
key: "target",
|
||||||
|
header: "to",
|
||||||
|
render: (d) =>
|
||||||
|
d.kind === "request_reviewer" ? `review by ${d.target}` : d.target,
|
||||||
|
filterValue: (d) => d.target,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
key: "outcome",
|
||||||
|
header: "outcome",
|
||||||
|
render: (d) => (
|
||||||
|
<span class={d.ok ? "" : "notify-failed"}>{outcomeText(d)}</span>
|
||||||
|
),
|
||||||
|
cellClass: "ui-table-prose",
|
||||||
|
sortBy: (d) => (d.ok ? 0 : 1),
|
||||||
|
},
|
||||||
|
];
|
||||||
|
|
||||||
|
export function Deliveries({ rows }: { rows: Delivery[] }) {
|
||||||
|
if (rows.length === 0) return <p>no deliveries yet</p>;
|
||||||
|
const placed = rows.map((d, place) => ({ ...d, place }));
|
||||||
|
return (
|
||||||
|
<Table
|
||||||
|
columns={COLUMNS}
|
||||||
|
rows={placed}
|
||||||
|
rowKey={(d) => String(d.place)}
|
||||||
|
storageKey="swarm-ui:notifications:deliveries-filters"
|
||||||
|
/>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
@ -249,3 +249,26 @@
|
||||||
.notify-name {
|
.notify-name {
|
||||||
flex: 0 1 24em;
|
flex: 0 1 24em;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* The dry run's counts beside each node, and the router's "not available". */
|
||||||
|
.notify-node-hits {
|
||||||
|
margin-left: auto;
|
||||||
|
flex: none;
|
||||||
|
color: var(--muted);
|
||||||
|
font-variant-numeric: tabular-nums;
|
||||||
|
}
|
||||||
|
|
||||||
|
.notify-dry-run {
|
||||||
|
margin: 0;
|
||||||
|
color: var(--muted);
|
||||||
|
font-size: 0.9em;
|
||||||
|
}
|
||||||
|
|
||||||
|
.notify-unavailable {
|
||||||
|
margin: 0;
|
||||||
|
color: var(--muted);
|
||||||
|
}
|
||||||
|
|
||||||
|
.notify-failed {
|
||||||
|
color: var(--red);
|
||||||
|
}
|
||||||
|
|
|
||||||
|
|
@ -19,6 +19,7 @@ import {
|
||||||
type RefreshIntervalMs,
|
type RefreshIntervalMs,
|
||||||
} from "../../ui/refresh-interval/RefreshInterval.js";
|
} from "../../ui/refresh-interval/RefreshInterval.js";
|
||||||
import { fetchJson, usePolled } from "../../ui/polled/usePolled.js";
|
import { fetchJson, usePolled } from "../../ui/polled/usePolled.js";
|
||||||
|
import { Deliveries } from "./Deliveries.js";
|
||||||
import {
|
import {
|
||||||
duplicateFlow,
|
duplicateFlow,
|
||||||
newFlow,
|
newFlow,
|
||||||
|
|
@ -32,11 +33,21 @@ import {
|
||||||
type Flow,
|
type Flow,
|
||||||
type FlowsView,
|
type FlowsView,
|
||||||
} from "./FlowTypes.js";
|
} from "./FlowTypes.js";
|
||||||
|
import {
|
||||||
|
DELIVERIES_URL,
|
||||||
|
DRY_RUN_URL,
|
||||||
|
hitsOf,
|
||||||
|
unavailableText,
|
||||||
|
type Deliveries as DeliveriesView,
|
||||||
|
type DryRun,
|
||||||
|
} from "./RouterLive.js";
|
||||||
import "./NotificationsPage.css";
|
import "./NotificationsPage.css";
|
||||||
|
|
||||||
const DEFAULT_INTERVAL_MS: RefreshIntervalMs = 30_000;
|
const DEFAULT_INTERVAL_MS: RefreshIntervalMs = 30_000;
|
||||||
|
|
||||||
function FlowTree({ flow }: { flow: Flow }) {
|
// `hits`, when the dry run covers this flow, puts the number of the newest events
|
||||||
|
// that reached each node beside it.
|
||||||
|
function FlowTree({ flow, hits }: { flow: Flow; hits?: Map<string, number> }) {
|
||||||
return (
|
return (
|
||||||
<ul class="notify-flow-tree">
|
<ul class="notify-flow-tree">
|
||||||
{treeRows(flow).map(({ node, depth }) => (
|
{treeRows(flow).map(({ node, depth }) => (
|
||||||
|
|
@ -47,6 +58,11 @@ function FlowTree({ flow }: { flow: Flow }) {
|
||||||
>
|
>
|
||||||
<span class="notify-node-kind">{node.kind}</span>
|
<span class="notify-node-kind">{node.kind}</span>
|
||||||
<span class="notify-node-text">{describeNode(node)}</span>
|
<span class="notify-node-text">{describeNode(node)}</span>
|
||||||
|
{hits ? (
|
||||||
|
<span class="notify-node-hits" title="events of the last dry run">
|
||||||
|
{hits.get(node.id) ?? 0}
|
||||||
|
</span>
|
||||||
|
) : null}
|
||||||
</li>
|
</li>
|
||||||
))}
|
))}
|
||||||
</ul>
|
</ul>
|
||||||
|
|
@ -63,6 +79,18 @@ export function NotificationsPage() {
|
||||||
} = usePolled(intervalMs, (signal) =>
|
} = usePolled(intervalMs, (signal) =>
|
||||||
fetchJson<FlowsView>("/api/notify/flows", signal),
|
fetchJson<FlowsView>("/api/notify/flows", signal),
|
||||||
);
|
);
|
||||||
|
// The router's own answers. Either is 503 while the running router predates
|
||||||
|
// it, which the page says in place of the section.
|
||||||
|
const { data: dry, error: dryError } = usePolled(intervalMs, (signal) =>
|
||||||
|
fetchJson<DryRun>(DRY_RUN_URL, signal),
|
||||||
|
);
|
||||||
|
const { data: deliveries, error: deliveriesError } = usePolled(
|
||||||
|
intervalMs,
|
||||||
|
(signal) => fetchJson<DeliveriesView>(DELIVERIES_URL, signal),
|
||||||
|
);
|
||||||
|
// A failed poll keeps the last answer; the counts are shown only while the
|
||||||
|
// latest poll worked.
|
||||||
|
const shown = dryError ? null : dry;
|
||||||
const [pending, setPending] = useState(false);
|
const [pending, setPending] = useState(false);
|
||||||
const [actionError, setActionError] = useState<ProblemDetails | null>(null);
|
const [actionError, setActionError] = useState<ProblemDetails | null>(null);
|
||||||
const [newName, setNewName] = useState("");
|
const [newName, setNewName] = useState("");
|
||||||
|
|
@ -149,6 +177,15 @@ export function NotificationsPage() {
|
||||||
{!error && view === null ? <p>loading…</p> : null}
|
{!error && view === null ? <p>loading…</p> : null}
|
||||||
{view ? (
|
{view ? (
|
||||||
<div class="notify-flows">
|
<div class="notify-flows">
|
||||||
|
{dryError ? (
|
||||||
|
<p class="notify-unavailable">
|
||||||
|
{unavailableText("hit counts", dryError)}
|
||||||
|
</p>
|
||||||
|
) : dry ? (
|
||||||
|
<p class="notify-dry-run">
|
||||||
|
counts: events that reached each node, of the newest {dry.events}
|
||||||
|
</p>
|
||||||
|
) : null}
|
||||||
<h3>built-in flows</h3>
|
<h3>built-in flows</h3>
|
||||||
{view.builtins.map((flow) => {
|
{view.builtins.map((flow) => {
|
||||||
const on = !view.disabled_builtins.includes(flow.id);
|
const on = !view.disabled_builtins.includes(flow.id);
|
||||||
|
|
@ -164,7 +201,7 @@ export function NotificationsPage() {
|
||||||
onChange={(checked) => setBuiltin(flow.id, checked)}
|
onChange={(checked) => setBuiltin(flow.id, checked)}
|
||||||
/>
|
/>
|
||||||
</div>
|
</div>
|
||||||
<FlowTree flow={flow} />
|
<FlowTree flow={flow} hits={hitsOf(shown, flow.id)} />
|
||||||
<Button disabled={pending} onClick={() => void duplicate(flow)}>
|
<Button disabled={pending} onClick={() => void duplicate(flow)}>
|
||||||
duplicate to edit
|
duplicate to edit
|
||||||
</Button>
|
</Button>
|
||||||
|
|
@ -202,9 +239,19 @@ export function NotificationsPage() {
|
||||||
duplicate
|
duplicate
|
||||||
</Button>
|
</Button>
|
||||||
</div>
|
</div>
|
||||||
<FlowTree flow={flow} />
|
<FlowTree flow={flow} hits={hitsOf(shown, flow.id)} />
|
||||||
</Card>
|
</Card>
|
||||||
))}
|
))}
|
||||||
|
<h3>recent deliveries</h3>
|
||||||
|
{deliveriesError ? (
|
||||||
|
<p class="notify-unavailable">
|
||||||
|
{unavailableText("the deliveries log", deliveriesError)}
|
||||||
|
</p>
|
||||||
|
) : deliveries ? (
|
||||||
|
<Deliveries rows={deliveries.rows} />
|
||||||
|
) : (
|
||||||
|
<p>loading…</p>
|
||||||
|
)}
|
||||||
</div>
|
</div>
|
||||||
) : null}
|
) : null}
|
||||||
</Panel>
|
</Panel>
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,62 @@
|
||||||
|
// Wire types and helpers for what the notification router reports about
|
||||||
|
// itself: its deliveries log and a dry run of the saved flows. Backed by
|
||||||
|
// swarm-controller's `GET /api/notify/deliveries` and
|
||||||
|
// `GET /api/notify/dry-run`, which answer 503 while the running router
|
||||||
|
// predates them.
|
||||||
|
import type { ProblemDetails } from "@hive/shared/api-error.js";
|
||||||
|
|
||||||
|
export const DELIVERIES_URL = "/api/notify/deliveries";
|
||||||
|
export const DRY_RUN_URL = "/api/notify/dry-run";
|
||||||
|
|
||||||
|
export interface Delivery {
|
||||||
|
// Unix seconds.
|
||||||
|
at: number;
|
||||||
|
flow: string;
|
||||||
|
node: string;
|
||||||
|
event: string;
|
||||||
|
repo: string;
|
||||||
|
number: number;
|
||||||
|
kind: "todo" | "request_reviewer";
|
||||||
|
target: string;
|
||||||
|
ok: boolean;
|
||||||
|
detail: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface Deliveries {
|
||||||
|
rows: Delivery[];
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface DryRun {
|
||||||
|
// How many of the newest forge events were run.
|
||||||
|
events: number;
|
||||||
|
flows: { flow: string; nodes: { node: string; hits: number }[] }[];
|
||||||
|
}
|
||||||
|
|
||||||
|
// Hits per node id, for one flow; `undefined` when the dry run has no entry
|
||||||
|
// for it (the flow is switched off, or there is no dry run).
|
||||||
|
export function hitsOf(
|
||||||
|
dry: DryRun | null,
|
||||||
|
flowId: string,
|
||||||
|
): Map<string, number> | undefined {
|
||||||
|
const flow = dry?.flows.find((f) => f.flow === flowId);
|
||||||
|
return flow && new Map(flow.nodes.map((n) => [n.node, n.hits]));
|
||||||
|
}
|
||||||
|
|
||||||
|
// `repo#n`, the way the forge names an issue or a PR.
|
||||||
|
export function issueRef(d: Pick<Delivery, "repo" | "number">): string {
|
||||||
|
return `${d.repo}#${d.number}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
// The outcome column: "ok" with what was done, or the error.
|
||||||
|
export function outcomeText(d: Pick<Delivery, "ok" | "detail">): string {
|
||||||
|
if (!d.ok) return `error: ${d.detail}`;
|
||||||
|
return d.detail === "" ? "ok" : `ok: ${d.detail}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
// The line shown in place of a section the router cannot serve.
|
||||||
|
export function unavailableText(what: string, problem: ProblemDetails): string {
|
||||||
|
const why = problem.detail ?? problem.title ?? "no reason given";
|
||||||
|
return problem.status === 503
|
||||||
|
? `${what} not available: ${why}`
|
||||||
|
: `${what} failed to load: ${why}`;
|
||||||
|
}
|
||||||
65
frontend/packages/swarm-ui/test/router-live.test.mjs
Normal file
65
frontend/packages/swarm-ui/test/router-live.test.mjs
Normal file
|
|
@ -0,0 +1,65 @@
|
||||||
|
// The notifications page's helpers for the router's deliveries and dry run.
|
||||||
|
// Run by `npm test`.
|
||||||
|
import assert from "node:assert/strict";
|
||||||
|
import { test } from "node:test";
|
||||||
|
import {
|
||||||
|
hitsOf,
|
||||||
|
issueRef,
|
||||||
|
outcomeText,
|
||||||
|
unavailableText,
|
||||||
|
} from "../src/pages/notifications/RouterLive.ts";
|
||||||
|
|
||||||
|
const dry = {
|
||||||
|
events: 50,
|
||||||
|
flows: [
|
||||||
|
{
|
||||||
|
flow: "assigned",
|
||||||
|
nodes: [
|
||||||
|
{ node: "source", hits: 49 },
|
||||||
|
{ node: "sink", hits: 0 },
|
||||||
|
],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
};
|
||||||
|
|
||||||
|
test("hitsOf maps a flow's node ids to their counts, zero included", () => {
|
||||||
|
const hits = hitsOf(dry, "assigned");
|
||||||
|
assert.equal(hits.get("source"), 49);
|
||||||
|
assert.equal(hits.get("sink"), 0);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("hitsOf has nothing for a flow the dry run left out, or no dry run", () => {
|
||||||
|
assert.equal(hitsOf(dry, "closed"), undefined);
|
||||||
|
assert.equal(hitsOf(null, "assigned"), undefined);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a delivery reads as its issue and its outcome", () => {
|
||||||
|
assert.equal(issueRef({ repo: "o/r", number: 9 }), "o/r#9");
|
||||||
|
assert.equal(
|
||||||
|
outcomeText({ ok: true, detail: "upsert forge:o/r#9:assignment" }),
|
||||||
|
"ok: upsert forge:o/r#9:assignment",
|
||||||
|
);
|
||||||
|
assert.equal(outcomeText({ ok: true, detail: "" }), "ok");
|
||||||
|
assert.equal(
|
||||||
|
outcomeText({ ok: false, detail: "the forge said no" }),
|
||||||
|
"error: the forge said no",
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a 503 reads as not available, any other failure as failed to load", () => {
|
||||||
|
assert.equal(
|
||||||
|
unavailableText("the deliveries log", {
|
||||||
|
status: 503,
|
||||||
|
detail: "the notification router does not answer this",
|
||||||
|
}),
|
||||||
|
"the deliveries log not available: the notification router does not answer this",
|
||||||
|
);
|
||||||
|
assert.equal(
|
||||||
|
unavailableText("hit counts", { status: 502, detail: "bad" }),
|
||||||
|
"hit counts failed to load: bad",
|
||||||
|
);
|
||||||
|
assert.equal(
|
||||||
|
unavailableText("hit counts", { status: 503 }),
|
||||||
|
"hit counts not available: no reason given",
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
@ -85,7 +85,8 @@ The swarm's control plane. The swarm UI and `swarmctl` are its clients.
|
||||||
- **Notification flows** — which of the notification router's built-in flows are
|
- **Notification flows** — which of the notification router's built-in flows are
|
||||||
switched off, and the custom flows, as one document in the `notify-flows`
|
switched off, and the custom flows, as one document in the `notify-flows`
|
||||||
KV bucket, which this daemon alone writes (`GET`/`PUT /api/notify/flows`);
|
KV bucket, which this daemon alone writes (`GET`/`PUT /api/notify/flows`);
|
||||||
the router reads it →
|
the router reads it. `GET /api/notify/deliveries` and `GET /api/notify/dry-run` ask
|
||||||
|
the router for its deliveries log and a dry run of the saved flows →
|
||||||
[`docs/swarm/notify-router.md`](../docs/swarm/notify-router.md#flows).
|
[`docs/swarm/notify-router.md`](../docs/swarm/notify-router.md#flows).
|
||||||
- **Operator profiles** — each operator's name and the pronouns on their
|
- **Operator profiles** — each operator's name and the pronouns on their
|
||||||
forge profile, for every member of authelia's operator group, as one list
|
forge profile, for every member of authelia's operator group, as one list
|
||||||
|
|
|
||||||
|
|
@ -52,6 +52,7 @@ mod matrix_account;
|
||||||
mod message_stream;
|
mod message_stream;
|
||||||
mod meta_flake;
|
mod meta_flake;
|
||||||
mod notify_flows;
|
mod notify_flows;
|
||||||
|
mod notify_router_api;
|
||||||
mod otel_http_client;
|
mod otel_http_client;
|
||||||
mod pulls;
|
mod pulls;
|
||||||
mod queue_identity;
|
mod queue_identity;
|
||||||
|
|
|
||||||
|
|
@ -76,7 +76,7 @@ impl FlowsView {
|
||||||
}
|
}
|
||||||
|
|
||||||
/// What the key holds.
|
/// What the key holds.
|
||||||
enum Stored {
|
pub(crate) enum Stored {
|
||||||
/// No key, or a deleted one.
|
/// No key, or a deleted one.
|
||||||
Nothing,
|
Nothing,
|
||||||
Document(Document),
|
Document(Document),
|
||||||
|
|
@ -85,7 +85,7 @@ enum Stored {
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The stored document, with the revision to write against.
|
/// The stored document, with the revision to write against.
|
||||||
async fn stored(store: &Store) -> anyhow::Result<(Stored, Option<u64>)> {
|
pub(crate) async fn stored(store: &Store) -> anyhow::Result<(Stored, Option<u64>)> {
|
||||||
struct Read(Stored);
|
struct Read(Stored);
|
||||||
impl Default for Read {
|
impl Default for Read {
|
||||||
fn default() -> Self {
|
fn default() -> Self {
|
||||||
|
|
@ -103,7 +103,7 @@ async fn stored(store: &Store) -> anyhow::Result<(Stored, Option<u64>)> {
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Open [`notify_flows::BUCKET`], creating it if nothing has yet.
|
/// Open [`notify_flows::BUCKET`], creating it if nothing has yet.
|
||||||
async fn open(client: &async_nats::Client) -> anyhow::Result<Store> {
|
pub(crate) async fn open(client: &async_nats::Client) -> anyhow::Result<Store> {
|
||||||
let js = async_nats::jetstream::new(client.clone());
|
let js = async_nats::jetstream::new(client.clone());
|
||||||
if let Ok(store) = js.get_key_value(notify_flows::BUCKET).await {
|
if let Ok(store) = js.get_key_value(notify_flows::BUCKET).await {
|
||||||
return Ok(store);
|
return Ok(store);
|
||||||
|
|
@ -119,7 +119,7 @@ async fn open(client: &async_nats::Client) -> anyhow::Result<Store> {
|
||||||
.with_context(|| format!("creating the {} bucket", notify_flows::BUCKET))
|
.with_context(|| format!("creating the {} bucket", notify_flows::BUCKET))
|
||||||
}
|
}
|
||||||
|
|
||||||
fn queue_problem(e: &anyhow::Error) -> ProblemDetails {
|
pub(crate) fn queue_problem(e: &anyhow::Error) -> ProblemDetails {
|
||||||
tracing::warn!(error = %format!("{e:#}"), "notify flows: queue failure");
|
tracing::warn!(error = %format!("{e:#}"), "notify flows: queue failure");
|
||||||
error_problem(wanted_error_status(e), &format!("{e:#}"))
|
error_problem(wanted_error_status(e), &format!("{e:#}"))
|
||||||
}
|
}
|
||||||
|
|
@ -158,7 +158,7 @@ pub(crate) async fn get_flows(
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
fn unreadable(why: &DecodeError) -> ProblemDetails {
|
pub(crate) fn unreadable(why: &DecodeError) -> ProblemDetails {
|
||||||
error_problem(
|
error_problem(
|
||||||
StatusCode::CONFLICT,
|
StatusCode::CONFLICT,
|
||||||
&format!("the stored flows document is not one this build reads: {why}"),
|
&format!("the stored flows document is not one this build reads: {why}"),
|
||||||
|
|
|
||||||
335
swarm-controller/src/notify_router_api.rs
Normal file
335
swarm-controller/src/notify_router_api.rs
Normal file
|
|
@ -0,0 +1,335 @@
|
||||||
|
//! `GET /api/notify/deliveries` and `GET /api/notify/dry-run`: what the
|
||||||
|
//! notification router has delivered, and what the saved flows would do with
|
||||||
|
//! the newest forge events.
|
||||||
|
//!
|
||||||
|
//! The router keeps the log and the events, so each is a NATS request to it
|
||||||
|
//! (`swarm_wire::router_api`). A router that predates them, or none at all,
|
||||||
|
//! has no subscriber: the queue answers "no responders" at once and these
|
||||||
|
//! answer 503 saying so, as they do when it hears the request and stays
|
||||||
|
//! silent. The saved document is what `GET /api/notify/flows` shows; a draft
|
||||||
|
//! is run by sending the same request with another document.
|
||||||
|
|
||||||
|
use axum::Json;
|
||||||
|
use axum::extract::{Query, State};
|
||||||
|
use axum::http::StatusCode;
|
||||||
|
use problem_details::ProblemDetails;
|
||||||
|
use serde::Deserialize;
|
||||||
|
use serde::de::DeserializeOwned;
|
||||||
|
use swarm_wire::notify_flows::Document;
|
||||||
|
use swarm_wire::router_api::{
|
||||||
|
DELIVERIES_SUBJECT, DRY_RUN_SUBJECT, Deliveries, DeliveriesRequest, DryRun, DryRunRequest,
|
||||||
|
REPLY_DEADLINE, Refused,
|
||||||
|
};
|
||||||
|
use utoipa::IntoParams;
|
||||||
|
|
||||||
|
use crate::notify_flows::{Stored, open, queue_problem, stored, unreadable};
|
||||||
|
use crate::routes::{AppState, error_problem};
|
||||||
|
use crate::schedules::{queue, refused};
|
||||||
|
|
||||||
|
/// Ask the router on `subject` and decode its answer as `T`.
|
||||||
|
///
|
||||||
|
/// 503: nothing hears the request, or what hears it does not answer in time;
|
||||||
|
/// both mean the running router does not have this feature. 502: it answered
|
||||||
|
/// with a refusal or with something this build cannot read.
|
||||||
|
async fn ask<T: DeserializeOwned>(
|
||||||
|
client: &async_nats::Client,
|
||||||
|
subject: &str,
|
||||||
|
body: Vec<u8>,
|
||||||
|
) -> Result<T, ProblemDetails> {
|
||||||
|
use async_nats::RequestErrorKind;
|
||||||
|
|
||||||
|
let sent = async_nats::Request::new()
|
||||||
|
.payload(body.into())
|
||||||
|
.timeout(Some(REPLY_DEADLINE));
|
||||||
|
let reply = client
|
||||||
|
.send_request(subject.to_owned(), sent)
|
||||||
|
.await
|
||||||
|
.map_err(|e| {
|
||||||
|
let detail = match e.kind() {
|
||||||
|
RequestErrorKind::NoResponders => "the notification router does not answer this: \
|
||||||
|
it is not running, or it is an older build than this controller"
|
||||||
|
.to_owned(),
|
||||||
|
RequestErrorKind::TimedOut => format!(
|
||||||
|
"the notification router sent no answer within {}s",
|
||||||
|
REPLY_DEADLINE.as_secs()
|
||||||
|
),
|
||||||
|
_ => format!("asking the notification router failed: {e}"),
|
||||||
|
};
|
||||||
|
error_problem(StatusCode::SERVICE_UNAVAILABLE, &detail)
|
||||||
|
})?;
|
||||||
|
if let Ok(answer) = serde_json::from_slice::<T>(&reply.payload) {
|
||||||
|
return Ok(answer);
|
||||||
|
}
|
||||||
|
let detail = match serde_json::from_slice::<Refused>(&reply.payload) {
|
||||||
|
Ok(Refused { error }) => format!("the notification router refused: {error}"),
|
||||||
|
Err(e) => format!("the notification router's answer did not decode: {e}"),
|
||||||
|
};
|
||||||
|
Err(error_problem(StatusCode::BAD_GATEWAY, &detail))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// `get_deliveries`'s query.
|
||||||
|
#[derive(Deserialize, IntoParams)]
|
||||||
|
pub struct DeliveriesQuery {
|
||||||
|
/// At most this many rows; 100 when absent, and the log keeps 500.
|
||||||
|
limit: Option<usize>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The newest deliveries the router made, newest first.
|
||||||
|
#[utoipa::path(
|
||||||
|
get,
|
||||||
|
path = "/api/notify/deliveries",
|
||||||
|
params(DeliveriesQuery),
|
||||||
|
responses(
|
||||||
|
(status = 200, description = "the router's deliveries log, newest first: when, which \
|
||||||
|
flow and node, the event and issue, who got it, and whether it went through",
|
||||||
|
body = Deliveries),
|
||||||
|
(status = 502, description = "the router refused, or its answer did not decode \
|
||||||
|
(problem+json)", body = String),
|
||||||
|
(status = 503, description = "no swarm queue is configured or connected here, or the \
|
||||||
|
router does not answer: it is not running, or it predates the log (problem+json)",
|
||||||
|
body = String),
|
||||||
|
),
|
||||||
|
tag = "notify"
|
||||||
|
)]
|
||||||
|
pub(crate) async fn get_deliveries(
|
||||||
|
State(state): State<AppState>,
|
||||||
|
Query(query): Query<DeliveriesQuery>,
|
||||||
|
) -> Result<Json<Deliveries>, ProblemDetails> {
|
||||||
|
let client = queue(&state).map_err(refused)?;
|
||||||
|
let body =
|
||||||
|
serde_json::to_vec(&DeliveriesRequest { limit: query.limit }).expect("a request encodes");
|
||||||
|
ask(&client, DELIVERIES_SUBJECT, body).await.map(Json)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The saved flows run over the newest 50 forge events, delivering nothing:
|
||||||
|
/// how many events reached each node.
|
||||||
|
#[utoipa::path(
|
||||||
|
get,
|
||||||
|
path = "/api/notify/dry-run",
|
||||||
|
responses(
|
||||||
|
(status = 200, description = "per flow that would run, the number of the newest 50 \
|
||||||
|
events that reached each node", body = DryRun),
|
||||||
|
(status = 409, description = "the stored document is not one this build reads \
|
||||||
|
(problem+json)", body = String),
|
||||||
|
(status = 502, description = "the router refused, or its answer did not decode \
|
||||||
|
(problem+json)", body = String),
|
||||||
|
(status = 503, description = "no swarm queue is configured or connected here, or the \
|
||||||
|
router does not answer: it is not running, or it predates the dry run \
|
||||||
|
(problem+json)", body = String),
|
||||||
|
(status = 500, description = "the saved flows could not be read (problem+json)",
|
||||||
|
body = String),
|
||||||
|
),
|
||||||
|
tag = "notify"
|
||||||
|
)]
|
||||||
|
pub(crate) async fn get_dry_run(
|
||||||
|
State(state): State<AppState>,
|
||||||
|
) -> Result<Json<DryRun>, ProblemDetails> {
|
||||||
|
let client = queue(&state).map_err(refused)?;
|
||||||
|
let store = open(&client).await.map_err(|e| queue_problem(&e))?;
|
||||||
|
let (saved, _) = stored(&store).await.map_err(|e| queue_problem(&e))?;
|
||||||
|
let document = match saved {
|
||||||
|
Stored::Nothing => Document::default(),
|
||||||
|
Stored::Document(doc) => doc,
|
||||||
|
Stored::Unreadable(e) => return Err(unreadable(&e)),
|
||||||
|
};
|
||||||
|
let body = serde_json::to_vec(&DryRunRequest { document }).expect("a request encodes");
|
||||||
|
ask(&client, DRY_RUN_SUBJECT, body).await.map(Json)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use futures_util::StreamExt as _;
|
||||||
|
use swarm_wire::notify_flows::{self, Document};
|
||||||
|
use swarm_wire::router_api::{FlowHits, NodeHits};
|
||||||
|
|
||||||
|
use super::*;
|
||||||
|
use crate::test_nats::TestNats;
|
||||||
|
|
||||||
|
fn query(limit: Option<usize>) -> Query<DeliveriesQuery> {
|
||||||
|
Query(DeliveriesQuery { limit })
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Answer the next request on `subject` with `reply`, handing the request
|
||||||
|
/// body back.
|
||||||
|
async fn answer(
|
||||||
|
client: &async_nats::Client,
|
||||||
|
subject: &'static str,
|
||||||
|
reply: Vec<u8>,
|
||||||
|
) -> tokio::task::JoinHandle<Vec<u8>> {
|
||||||
|
let mut requests = client.subscribe(subject).await.expect("subscribed");
|
||||||
|
client.flush().await.expect("flushed");
|
||||||
|
let client = client.clone();
|
||||||
|
tokio::spawn(async move {
|
||||||
|
let m = requests.next().await.expect("a request");
|
||||||
|
client
|
||||||
|
.publish(m.reply.expect("a reply subject"), reply.into())
|
||||||
|
.await
|
||||||
|
.expect("answered");
|
||||||
|
m.payload.to_vec()
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A router that predates the log and the dry run has no subscriber: both
|
||||||
|
/// answer 503 with the reason, not 500 and not a hang. The control is the
|
||||||
|
/// same two calls with a router answering.
|
||||||
|
#[tokio::test]
|
||||||
|
async fn without_a_router_that_answers_both_are_unavailable() {
|
||||||
|
let nats = TestNats::start();
|
||||||
|
let client = nats.client().await;
|
||||||
|
let state = crate::agent_limits::tests::state(&client);
|
||||||
|
|
||||||
|
let deliveries = get_deliveries(State(state.clone()), query(None))
|
||||||
|
.await
|
||||||
|
.expect_err("nobody answers");
|
||||||
|
assert_eq!(deliveries.status, Some(StatusCode::SERVICE_UNAVAILABLE));
|
||||||
|
assert!(
|
||||||
|
deliveries
|
||||||
|
.detail
|
||||||
|
.as_deref()
|
||||||
|
.unwrap_or_default()
|
||||||
|
.contains("older build"),
|
||||||
|
"{deliveries:?}"
|
||||||
|
);
|
||||||
|
let dry = get_dry_run(State(state.clone()))
|
||||||
|
.await
|
||||||
|
.expect_err("nobody answers");
|
||||||
|
assert_eq!(dry.status, Some(StatusCode::SERVICE_UNAVAILABLE));
|
||||||
|
assert!(
|
||||||
|
dry.detail
|
||||||
|
.as_deref()
|
||||||
|
.unwrap_or_default()
|
||||||
|
.contains("older build"),
|
||||||
|
"{dry:?}"
|
||||||
|
);
|
||||||
|
|
||||||
|
// Control: with a router answering, the same calls are 200.
|
||||||
|
let rows = answer(&client, DELIVERIES_SUBJECT, br#"{"rows":[]}"#.to_vec()).await;
|
||||||
|
let Json(got) = get_deliveries(State(state.clone()), query(None))
|
||||||
|
.await
|
||||||
|
.expect("answered");
|
||||||
|
assert!(got.rows.is_empty());
|
||||||
|
rows.await.expect("the router task");
|
||||||
|
let hits = DryRun {
|
||||||
|
events: 0,
|
||||||
|
flows: Vec::new(),
|
||||||
|
};
|
||||||
|
let dry_run = answer(&client, DRY_RUN_SUBJECT, serde_json::to_vec(&hits).unwrap()).await;
|
||||||
|
let Json(got) = get_dry_run(State(state)).await.expect("answered");
|
||||||
|
assert_eq!(got, hits);
|
||||||
|
dry_run.await.expect("the router task");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// No queue on this host is 503 too.
|
||||||
|
#[tokio::test]
|
||||||
|
async fn without_a_queue_both_are_unavailable() {
|
||||||
|
let (state, _) = crate::routes::tests::state_with_roster();
|
||||||
|
let a = get_deliveries(State(state.clone()), query(None))
|
||||||
|
.await
|
||||||
|
.expect_err("no queue");
|
||||||
|
let b = get_dry_run(State(state)).await.expect_err("no queue");
|
||||||
|
assert_eq!(a.status, Some(StatusCode::SERVICE_UNAVAILABLE));
|
||||||
|
assert_eq!(b.status, Some(StatusCode::SERVICE_UNAVAILABLE));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The limit reaches the router, and the dry run sends the saved document.
|
||||||
|
#[tokio::test]
|
||||||
|
async fn the_limit_and_the_saved_document_reach_the_router() {
|
||||||
|
let nats = TestNats::start();
|
||||||
|
let client = nats.client().await;
|
||||||
|
let state = crate::agent_limits::tests::state(&client);
|
||||||
|
|
||||||
|
let asked = answer(&client, DELIVERIES_SUBJECT, br#"{"rows":[]}"#.to_vec()).await;
|
||||||
|
let _ = get_deliveries(State(state.clone()), query(Some(7)))
|
||||||
|
.await
|
||||||
|
.expect("answered");
|
||||||
|
let request: DeliveriesRequest =
|
||||||
|
serde_json::from_slice(&asked.await.expect("task")).expect("a request");
|
||||||
|
assert_eq!(request.limit, Some(7));
|
||||||
|
|
||||||
|
// Nothing saved: the default document.
|
||||||
|
let reply = DryRun {
|
||||||
|
events: 3,
|
||||||
|
flows: vec![FlowHits {
|
||||||
|
flow: "assigned".to_owned(),
|
||||||
|
nodes: vec![NodeHits {
|
||||||
|
node: "source".to_owned(),
|
||||||
|
hits: 3,
|
||||||
|
}],
|
||||||
|
}],
|
||||||
|
};
|
||||||
|
let asked = answer(
|
||||||
|
&client,
|
||||||
|
DRY_RUN_SUBJECT,
|
||||||
|
serde_json::to_vec(&reply).unwrap(),
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
let Json(got) = get_dry_run(State(state.clone())).await.expect("answered");
|
||||||
|
assert_eq!(got, reply);
|
||||||
|
let sent: DryRunRequest =
|
||||||
|
serde_json::from_slice(&asked.await.expect("task")).expect("a request");
|
||||||
|
assert_eq!(sent.document, Document::default());
|
||||||
|
|
||||||
|
// A saved document: the router gets that one.
|
||||||
|
let store = open(&client).await.expect("bucket");
|
||||||
|
let saved = Document {
|
||||||
|
disabled_builtins: vec!["closed".to_owned()],
|
||||||
|
..Document::default()
|
||||||
|
};
|
||||||
|
store
|
||||||
|
.put(
|
||||||
|
notify_flows::KEY,
|
||||||
|
serde_json::to_vec(&saved).unwrap().into(),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.expect("seed");
|
||||||
|
let asked = answer(
|
||||||
|
&client,
|
||||||
|
DRY_RUN_SUBJECT,
|
||||||
|
serde_json::to_vec(&reply).unwrap(),
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
let _ = get_dry_run(State(state)).await.expect("answered");
|
||||||
|
let sent: DryRunRequest =
|
||||||
|
serde_json::from_slice(&asked.await.expect("task")).expect("a request");
|
||||||
|
assert_eq!(sent.document, saved);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A refusal from the router is 502 with its reason; an unreadable saved
|
||||||
|
/// document is 409 and never reaches the router.
|
||||||
|
#[tokio::test]
|
||||||
|
async fn a_refusal_is_502_and_an_unreadable_document_is_409() {
|
||||||
|
let nats = TestNats::start();
|
||||||
|
let client = nats.client().await;
|
||||||
|
let state = crate::agent_limits::tests::state(&client);
|
||||||
|
|
||||||
|
let router = answer(
|
||||||
|
&client,
|
||||||
|
DRY_RUN_SUBJECT,
|
||||||
|
br#"{"error":"no events"}"#.to_vec(),
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
let refused = get_dry_run(State(state.clone()))
|
||||||
|
.await
|
||||||
|
.expect_err("refused");
|
||||||
|
assert_eq!(refused.status, Some(StatusCode::BAD_GATEWAY));
|
||||||
|
assert!(
|
||||||
|
refused
|
||||||
|
.detail
|
||||||
|
.as_deref()
|
||||||
|
.unwrap_or_default()
|
||||||
|
.contains("no events")
|
||||||
|
);
|
||||||
|
router.await.expect("the router task");
|
||||||
|
|
||||||
|
let store = open(&client).await.expect("bucket");
|
||||||
|
store
|
||||||
|
.put(
|
||||||
|
notify_flows::KEY,
|
||||||
|
br#"{"version":2,"flows":[]}"#.to_vec().into(),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.expect("seed");
|
||||||
|
let unreadable = get_dry_run(State(state)).await.expect_err("version 2");
|
||||||
|
assert_eq!(unreadable.status, Some(StatusCode::CONFLICT));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -31,8 +31,9 @@ use crate::{
|
||||||
agent_control, agent_icon, agent_input, agent_limits, agent_login, agent_permissions,
|
agent_control, agent_icon, agent_input, agent_limits, agent_login, agent_permissions,
|
||||||
agent_renewal, agent_state_stream, agent_status, agents_md, alerts, archetypes, auth, builder,
|
agent_renewal, agent_state_stream, agent_status, agents_md, alerts, archetypes, auth, builder,
|
||||||
claude_login, config_pr, forge, forge_account, github_account, inference, issue_report,
|
claude_login, config_pr, forge, forge_account, github_account, inference, issue_report,
|
||||||
linked_accounts, matrix_account, message_stream, notify_flows, pulls, queue_identity,
|
linked_accounts, matrix_account, message_stream, notify_flows, notify_router_api, pulls,
|
||||||
review_classifier, schedules, status, subagent_term, term_stream, wanted, webhook,
|
queue_identity, review_classifier, schedules, status, subagent_term, term_stream, wanted,
|
||||||
|
webhook,
|
||||||
};
|
};
|
||||||
|
|
||||||
/// Where the daemon binds, overridable via `SWARM_CONTROLLER_SOCKET`.
|
/// Where the daemon binds, overridable via `SWARM_CONTROLLER_SOCKET`.
|
||||||
|
|
@ -643,6 +644,8 @@ pub(crate) fn build_app(state: AppState) -> axum::Router {
|
||||||
review_classifier::settings::put_settings
|
review_classifier::settings::put_settings
|
||||||
))
|
))
|
||||||
.routes(routes!(notify_flows::get_flows, notify_flows::put_flows))
|
.routes(routes!(notify_flows::get_flows, notify_flows::put_flows))
|
||||||
|
.routes(routes!(notify_router_api::get_deliveries))
|
||||||
|
.routes(routes!(notify_router_api::get_dry_run))
|
||||||
.routes(routes!(webhook::post_webhook_forge))
|
.routes(routes!(webhook::post_webhook_forge))
|
||||||
.split_for_parts();
|
.split_for_parts();
|
||||||
// Just the JSON, not the UI — Swagger UI itself is nginx-hosted from
|
// Just the JSON, not the UI — Swagger UI itself is nginx-hosted from
|
||||||
|
|
|
||||||
|
|
@ -298,10 +298,15 @@ impl Policy {
|
||||||
/// `swarm_wire::router_sink::REQUEST_REVIEWER_SUBJECT` and hear the answer
|
/// `swarm_wire::router_sink::REQUEST_REVIEWER_SUBJECT` and hear the answer
|
||||||
/// on its own inbox. The controller is the subject's one subscriber and
|
/// on its own inbox. The controller is the subject's one subscriber and
|
||||||
/// holds the forge token; no wildcard over `$SWARM.router.sink`.
|
/// holds the forge token; no wildcard over `$SWARM.router.sink`.
|
||||||
|
/// - **Deliveries log.** Open or create `NOTIFY_ROUTER_LOG`, publish its one
|
||||||
|
/// subject, and read rows and the newest events by `STREAM.MSG.GET`.
|
||||||
|
/// - **Api.** Hear the two `swarm_wire::router_api` requests and answer each
|
||||||
|
/// once, under `allow_responses`. The controller is their one publisher.
|
||||||
fn notify_router_permissions(client_id: &str) -> Permissions {
|
fn notify_router_permissions(client_id: &str) -> Permissions {
|
||||||
use swarm_wire::{forge_events, notify_flows, notify_router, router_todo};
|
use swarm_wire::{forge_events, notify_flows, notify_router, router_api, router_todo};
|
||||||
|
|
||||||
let events = forge_events::STREAM;
|
let events = forge_events::STREAM;
|
||||||
|
let log_stream = router_api::LOG_STREAM;
|
||||||
let consumer = forge_events::ROUTER_CONSUMER;
|
let consumer = forge_events::ROUTER_CONSUMER;
|
||||||
let state = format!("KV_{}", notify_router::STATE_BUCKET);
|
let state = format!("KV_{}", notify_router::STATE_BUCKET);
|
||||||
let flows = format!("KV_{}", notify_flows::BUCKET);
|
let flows = format!("KV_{}", notify_flows::BUCKET);
|
||||||
|
|
@ -326,12 +331,21 @@ impl Policy {
|
||||||
format!("$JS.API.DIRECT.GET.{flows}.>"),
|
format!("$JS.API.DIRECT.GET.{flows}.>"),
|
||||||
format!("$JS.API.STREAM.INFO.{}", Self::agent_status_stream()),
|
format!("$JS.API.STREAM.INFO.{}", Self::agent_status_stream()),
|
||||||
swarm_wire::router_sink::REQUEST_REVIEWER_SUBJECT.to_owned(),
|
swarm_wire::router_sink::REQUEST_REVIEWER_SUBJECT.to_owned(),
|
||||||
|
format!("$JS.API.STREAM.MSG.GET.{events}"),
|
||||||
|
format!("$JS.API.STREAM.INFO.{log_stream}"),
|
||||||
|
format!("$JS.API.STREAM.CREATE.{log_stream}"),
|
||||||
|
format!("$JS.API.STREAM.MSG.GET.{log_stream}"),
|
||||||
|
router_api::LOG_SUBJECT.to_owned(),
|
||||||
Self::inbox(client_id),
|
Self::inbox(client_id),
|
||||||
]);
|
]);
|
||||||
Permissions {
|
Permissions {
|
||||||
publish,
|
publish,
|
||||||
subscribe: Some(vec![Self::inbox(client_id)]),
|
subscribe: Some(vec![
|
||||||
responses: None,
|
router_api::DELIVERIES_SUBJECT.to_owned(),
|
||||||
|
router_api::DRY_RUN_SUBJECT.to_owned(),
|
||||||
|
Self::inbox(client_id),
|
||||||
|
]),
|
||||||
|
responses: Some(router_api::REPLY_DEADLINE),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -698,6 +712,12 @@ impl Policy {
|
||||||
swarm_wire::builder::LOCK_SUBJECT.to_owned(),
|
swarm_wire::builder::LOCK_SUBJECT.to_owned(),
|
||||||
// An operator's claude login step, sent as a request the same way.
|
// An operator's claude login step, sent as a request the same way.
|
||||||
swarm_wire::claude_login::SUBJECT.to_owned(),
|
swarm_wire::claude_login::SUBJECT.to_owned(),
|
||||||
|
// The notification router's deliveries log and a dry run of a flows
|
||||||
|
// document, sent as requests: the reply comes back on this client's
|
||||||
|
// inbox. The router is their one subscriber; no other grant names
|
||||||
|
// them.
|
||||||
|
swarm_wire::router_api::DELIVERIES_SUBJECT.to_owned(),
|
||||||
|
swarm_wire::router_api::DRY_RUN_SUBJECT.to_owned(),
|
||||||
// Verified forge deliveries, kept in the forge events stream for
|
// Verified forge deliveries, kept in the forge events stream for
|
||||||
// the notification router. This client is their one publisher: the
|
// the notification router. This client is their one publisher: the
|
||||||
// router reads them and no hive or agent grant names them.
|
// router reads them and no hive or agent grant names them.
|
||||||
|
|
@ -1748,7 +1768,9 @@ mod tests {
|
||||||
#[test]
|
#[test]
|
||||||
fn the_notify_router_drains_its_own_consumer_and_writes_only_its_own_state() {
|
fn the_notify_router_drains_its_own_consumer_and_writes_only_its_own_state() {
|
||||||
use swarm_queue_client::client_inbox_prefix;
|
use swarm_queue_client::client_inbox_prefix;
|
||||||
use swarm_wire::{forge_events, notify_flows, notify_router, router_sink, router_todo};
|
use swarm_wire::{
|
||||||
|
forge_events, notify_flows, notify_router, router_api, router_sink, router_todo,
|
||||||
|
};
|
||||||
|
|
||||||
let p = with_router()
|
let p = with_router()
|
||||||
.permissions(NOTIFY_ROUTER)
|
.permissions(NOTIFY_ROUTER)
|
||||||
|
|
@ -1771,6 +1793,13 @@ mod tests {
|
||||||
format!("$JS.API.DIRECT.GET.KV_{state}.$KV.{state}.42.5571"),
|
format!("$JS.API.DIRECT.GET.KV_{state}.$KV.{state}.42.5571"),
|
||||||
router_sink::REQUEST_REVIEWER_SUBJECT.to_owned(),
|
router_sink::REQUEST_REVIEWER_SUBJECT.to_owned(),
|
||||||
"$JS.API.STREAM.INFO.KV_agent-status".to_owned(),
|
"$JS.API.STREAM.INFO.KV_agent-status".to_owned(),
|
||||||
|
// The deliveries log: make it, append to it, read it by sequence.
|
||||||
|
format!("$JS.API.STREAM.CREATE.{}", router_api::LOG_STREAM),
|
||||||
|
format!("$JS.API.STREAM.INFO.{}", router_api::LOG_STREAM),
|
||||||
|
format!("$JS.API.STREAM.MSG.GET.{}", router_api::LOG_STREAM),
|
||||||
|
router_api::LOG_SUBJECT.to_owned(),
|
||||||
|
// The newest forge events, for a dry run.
|
||||||
|
format!("$JS.API.STREAM.MSG.GET.{events}"),
|
||||||
// The flows document: read it.
|
// The flows document: read it.
|
||||||
format!("$JS.API.STREAM.INFO.KV_{flows}"),
|
format!("$JS.API.STREAM.INFO.KV_{flows}"),
|
||||||
format!(
|
format!(
|
||||||
|
|
@ -1797,6 +1826,17 @@ mod tests {
|
||||||
"$KV.agent-schedules.atlas".to_owned(),
|
"$KV.agent-schedules.atlas".to_owned(),
|
||||||
"$SWARM.router.sink.other".to_owned(),
|
"$SWARM.router.sink.other".to_owned(),
|
||||||
"$SWARM.router.sink.>".to_owned(),
|
"$SWARM.router.sink.>".to_owned(),
|
||||||
|
// The log: no delete, no purge, no message delete; no read of the todo
|
||||||
|
// stream; no other subject below the router's two prefixes; and no
|
||||||
|
// sending of the requests it answers.
|
||||||
|
format!("$JS.API.STREAM.DELETE.{}", router_api::LOG_STREAM),
|
||||||
|
format!("$JS.API.STREAM.PURGE.{}", router_api::LOG_STREAM),
|
||||||
|
format!("$JS.API.STREAM.MSG.DELETE.{}", router_api::LOG_STREAM),
|
||||||
|
format!("$JS.API.STREAM.MSG.GET.{}", router_todo::STREAM),
|
||||||
|
"$SWARM.router.log.other".to_owned(),
|
||||||
|
"$SWARM.router.api.other".to_owned(),
|
||||||
|
router_api::DRY_RUN_SUBJECT.to_owned(),
|
||||||
|
router_api::DELIVERIES_SUBJECT.to_owned(),
|
||||||
// The flows document: not write it, nor make the bucket.
|
// The flows document: not write it, nor make the bucket.
|
||||||
format!("$KV.{flows}.{}", notify_flows::KEY),
|
format!("$KV.{flows}.{}", notify_flows::KEY),
|
||||||
format!("$JS.API.STREAM.CREATE.KV_{flows}"),
|
format!("$JS.API.STREAM.CREATE.KV_{flows}"),
|
||||||
|
|
@ -1807,8 +1847,15 @@ mod tests {
|
||||||
assert!(!may(&refused), "the router may reach {refused}");
|
assert!(!may(&refused), "the router may reach {refused}");
|
||||||
}
|
}
|
||||||
let inbox = client_inbox_prefix(NOTIFY_ROUTER);
|
let inbox = client_inbox_prefix(NOTIFY_ROUTER);
|
||||||
assert_eq!(p.subscribe, Some(vec![format!("{inbox}.>")]));
|
assert_eq!(
|
||||||
assert_eq!(p.responses, None);
|
p.subscribe,
|
||||||
|
Some(vec![
|
||||||
|
router_api::DELIVERIES_SUBJECT.to_owned(),
|
||||||
|
router_api::DRY_RUN_SUBJECT.to_owned(),
|
||||||
|
format!("{inbox}.>"),
|
||||||
|
])
|
||||||
|
);
|
||||||
|
assert_eq!(p.responses, Some(router_api::REPLY_DEADLINE));
|
||||||
assert_eq!(policy().permissions(NOTIFY_ROUTER), None);
|
assert_eq!(policy().permissions(NOTIFY_ROUTER), None);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -1854,6 +1901,48 @@ mod tests {
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The controller asks and the router answers: nobody else may send the
|
||||||
|
/// two requests, and nobody else may hear them.
|
||||||
|
#[test]
|
||||||
|
fn only_the_reader_may_ask_the_router_and_only_the_router_may_hear_it() {
|
||||||
|
use swarm_wire::router_api::{DELIVERIES_SUBJECT, DRY_RUN_SUBJECT, LOG_SUBJECT};
|
||||||
|
|
||||||
|
let p = with_router();
|
||||||
|
let grants = [
|
||||||
|
(
|
||||||
|
"the reader",
|
||||||
|
p.permissions("swarm-controller").expect("reader"),
|
||||||
|
),
|
||||||
|
("a hive", p.permissions("hive-alpha").expect("a hive")),
|
||||||
|
("the router", p.permissions(NOTIFY_ROUTER).expect("router")),
|
||||||
|
("the minter", p.permissions(CLAUDE_LOGIN).expect("minter")),
|
||||||
|
("the builder", p.permissions(BUILDER).expect("builder")),
|
||||||
|
(
|
||||||
|
"the classifier",
|
||||||
|
p.permissions(CLASSIFIER).expect("classifier"),
|
||||||
|
),
|
||||||
|
("atlas", p.agent_token_permissions("atlas").expect("atlas")),
|
||||||
|
];
|
||||||
|
for (who, grant) in &grants {
|
||||||
|
for subject in [DELIVERIES_SUBJECT, DRY_RUN_SUBJECT] {
|
||||||
|
let sends = grant.publish.iter().any(|g| covers(g, subject));
|
||||||
|
let hears = may_subscribe(grant, subject);
|
||||||
|
assert_eq!(sends, *who == "the reader", "{who} sends {subject}");
|
||||||
|
assert_eq!(hears, *who == "the router", "{who} hears {subject}");
|
||||||
|
}
|
||||||
|
// The log has one writer.
|
||||||
|
let appends = grant.publish.iter().any(|g| covers(g, LOG_SUBJECT));
|
||||||
|
assert_eq!(appends, *who == "the router", "{who} appends to the log");
|
||||||
|
}
|
||||||
|
// The router answers for as long as the controller waits.
|
||||||
|
let router = &grants[2].1;
|
||||||
|
assert!(
|
||||||
|
router.responses >= Some(swarm_wire::router_api::REPLY_DEADLINE),
|
||||||
|
"{:?}",
|
||||||
|
router.responses
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn only_the_router_may_queue_an_agents_todo_and_only_the_reader_may_keep_a_forge_event() {
|
fn only_the_router_may_queue_an_agents_todo_and_only_the_reader_may_keep_a_forge_event() {
|
||||||
use swarm_wire::{forge_events, router_todo};
|
use swarm_wire::{forge_events, router_todo};
|
||||||
|
|
|
||||||
|
|
@ -30,5 +30,9 @@ swarm-wire.workspace = true
|
||||||
tokio.workspace = true
|
tokio.workspace = true
|
||||||
tracing.workspace = true
|
tracing.workspace = true
|
||||||
|
|
||||||
|
# A fresh directory per test `nats-server`, for its store and ports file.
|
||||||
|
[dev-dependencies]
|
||||||
|
tempfile = "3"
|
||||||
|
|
||||||
[lints]
|
[lints]
|
||||||
workspace = true
|
workspace = true
|
||||||
|
|
|
||||||
|
|
@ -4,5 +4,8 @@ Routes forge events to agents' todo lists. It drains its consumer on the
|
||||||
forge events stream that swarm-controller fills from its webhooks, runs its
|
forge events stream that swarm-controller fills from its webhooks, runs its
|
||||||
flows over each delivery, and queues the todo ops they emit on each agent's
|
flows over each delivery, and queues the todo ops they emit on each agent's
|
||||||
subject of the `ROUTER_TODOS` stream. Each agent's harness applies them.
|
subject of the `ROUTER_TODOS` stream. Each agent's harness applies them.
|
||||||
|
It keeps the newest 500 deliveries in `NOTIFY_ROUTER_LOG` and answers
|
||||||
|
swarm-controller's requests for them and for a dry run of the flows over the
|
||||||
|
newest 50 events.
|
||||||
|
|
||||||
Design: [`docs/swarm/notify-router.md`](../docs/swarm/notify-router.md).
|
Design: [`docs/swarm/notify-router.md`](../docs/swarm/notify-router.md).
|
||||||
|
|
|
||||||
288
swarm-notify-router/src/api.rs
Normal file
288
swarm-notify-router/src/api.rs
Normal file
|
|
@ -0,0 +1,288 @@
|
||||||
|
//! The router's two answers to swarm-controller: the newest deliveries and a
|
||||||
|
//! dry run (`swarm_wire::router_api`).
|
||||||
|
//!
|
||||||
|
//! Each is a NATS request. A dry run replays the newest forge events from the
|
||||||
|
//! events stream through [`dry_run::run`] and delivers nothing, writes no
|
||||||
|
//! snapshot and sends no review request. The router hears these on their own
|
||||||
|
//! task, so a slow replay never holds up the events.
|
||||||
|
|
||||||
|
use anyhow::Result;
|
||||||
|
use async_nats::jetstream;
|
||||||
|
use futures_util::StreamExt as _;
|
||||||
|
use serde::Serialize;
|
||||||
|
use swarm_wire::forge_events;
|
||||||
|
use swarm_wire::router_api::{
|
||||||
|
DEFAULT_ROWS, DELIVERIES_SUBJECT, DRY_RUN_EVENTS, DRY_RUN_SUBJECT, Deliveries,
|
||||||
|
DeliveriesRequest, DryRunRequest, Refused,
|
||||||
|
};
|
||||||
|
|
||||||
|
use crate::{dry_run, log};
|
||||||
|
|
||||||
|
/// Answer requests on both subjects until the connection closes.
|
||||||
|
pub async fn serve(client: async_nats::Client) {
|
||||||
|
let js = jetstream::new(client.clone());
|
||||||
|
let (mut deliveries, mut dry) = match (
|
||||||
|
client.subscribe(DELIVERIES_SUBJECT).await,
|
||||||
|
client.subscribe(DRY_RUN_SUBJECT).await,
|
||||||
|
) {
|
||||||
|
(Ok(a), Ok(b)) => (a, b),
|
||||||
|
(a, b) => {
|
||||||
|
let why = a
|
||||||
|
.err()
|
||||||
|
.map(|e| e.to_string())
|
||||||
|
.or(b.err().map(|e| e.to_string()));
|
||||||
|
tracing::error!(error = ?why, "notify router: cannot hear api requests; the controller will report them unavailable");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
};
|
||||||
|
tracing::info!("notify router: answering deliveries and dry-run requests");
|
||||||
|
loop {
|
||||||
|
let (request, dry_run) = tokio::select! {
|
||||||
|
Some(m) = deliveries.next() => (m, false),
|
||||||
|
Some(m) = dry.next() => (m, true),
|
||||||
|
else => return,
|
||||||
|
};
|
||||||
|
let (client, js) = (client.clone(), js.clone());
|
||||||
|
tokio::spawn(async move {
|
||||||
|
let Some(reply) = request.reply else { return };
|
||||||
|
let answer = if dry_run {
|
||||||
|
dry_run_answer(&js, &request.payload).await
|
||||||
|
} else {
|
||||||
|
deliveries_answer(&js, &request.payload).await
|
||||||
|
};
|
||||||
|
if let Err(e) = client.publish(reply, answer.into()).await {
|
||||||
|
tracing::warn!(error = %e, "notify router: could not answer an api request");
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The reply body: `value`, or a [`Refused`] saying why not.
|
||||||
|
fn body<T: Serialize>(answer: Result<T>) -> Vec<u8> {
|
||||||
|
match answer {
|
||||||
|
Ok(value) => serde_json::to_vec(&value).expect("an answer encodes"),
|
||||||
|
Err(e) => {
|
||||||
|
tracing::warn!(
|
||||||
|
error = format!("{e:#}"),
|
||||||
|
"notify router: an api request failed"
|
||||||
|
);
|
||||||
|
serde_json::to_vec(&Refused {
|
||||||
|
error: format!("{e:#}"),
|
||||||
|
})
|
||||||
|
.expect("a refusal encodes")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn deliveries_answer(js: &jetstream::Context, payload: &[u8]) -> Vec<u8> {
|
||||||
|
body(
|
||||||
|
async {
|
||||||
|
let request: DeliveriesRequest = if payload.is_empty() {
|
||||||
|
DeliveriesRequest::default()
|
||||||
|
} else {
|
||||||
|
serde_json::from_slice(payload)?
|
||||||
|
};
|
||||||
|
let limit = request.limit.unwrap_or(DEFAULT_ROWS);
|
||||||
|
Ok(Deliveries {
|
||||||
|
rows: log::recent(js, limit).await?,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
.await,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn dry_run_answer(js: &jetstream::Context, payload: &[u8]) -> Vec<u8> {
|
||||||
|
body(
|
||||||
|
async {
|
||||||
|
let request: DryRunRequest = serde_json::from_slice(payload)?;
|
||||||
|
request
|
||||||
|
.document
|
||||||
|
.check()
|
||||||
|
.map_err(|why| anyhow::anyhow!("the document fails its check: {why}"))?;
|
||||||
|
let (flows, _) = request.document.active();
|
||||||
|
let mut events: Vec<(String, Vec<u8>)> =
|
||||||
|
log::newest(js, forge_events::STREAM, DRY_RUN_EVENTS)
|
||||||
|
.await?
|
||||||
|
.into_iter()
|
||||||
|
.filter_map(|(subject, body)| {
|
||||||
|
Some((forge_events::event_type(&subject)?.to_owned(), body))
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
events.reverse();
|
||||||
|
Ok(dry_run::run(&flows, &events))
|
||||||
|
}
|
||||||
|
.await,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use std::time::Duration;
|
||||||
|
|
||||||
|
use swarm_wire::notify_flows::Document;
|
||||||
|
use swarm_wire::router_api::{DryRun, Kind, LOG_STREAM, Row};
|
||||||
|
use swarm_wire::router_todo;
|
||||||
|
|
||||||
|
use super::*;
|
||||||
|
use crate::test_nats::TestNats;
|
||||||
|
|
||||||
|
/// Send a request to the router's api, retrying while its subscription
|
||||||
|
/// is not yet in place.
|
||||||
|
async fn ask(client: &async_nats::Client, subject: &str, body: Vec<u8>) -> Vec<u8> {
|
||||||
|
for _ in 0..50 {
|
||||||
|
if let Ok(m) = client
|
||||||
|
.request(subject.to_owned(), body.clone().into())
|
||||||
|
.await
|
||||||
|
{
|
||||||
|
return m.payload.to_vec();
|
||||||
|
}
|
||||||
|
tokio::time::sleep(Duration::from_millis(100)).await;
|
||||||
|
}
|
||||||
|
panic!("nothing answered {subject}");
|
||||||
|
}
|
||||||
|
|
||||||
|
fn row(n: u64) -> Row {
|
||||||
|
log::Cause {
|
||||||
|
event: "issue_assign".to_owned(),
|
||||||
|
repo: "o/r".to_owned(),
|
||||||
|
number: n,
|
||||||
|
}
|
||||||
|
.row(
|
||||||
|
"assigned",
|
||||||
|
"sink",
|
||||||
|
Kind::Todo,
|
||||||
|
"atlas",
|
||||||
|
Ok("upsert k".to_owned()),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn publish_event(js: &jetstream::Context, event: &str, n: i64) {
|
||||||
|
let body = serde_json::json!({
|
||||||
|
"action": "assigned",
|
||||||
|
"repository": { "full_name": "o/r", "id": 1 },
|
||||||
|
"issue": { "number": n, "title": "t", "html_url": "u", "assignees": [{ "login": "atlas" }] },
|
||||||
|
"sender": { "login": "mara" },
|
||||||
|
});
|
||||||
|
js.publish(
|
||||||
|
forge_events::subject(event),
|
||||||
|
serde_json::to_vec(&body).unwrap().into(),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.expect("published")
|
||||||
|
.await
|
||||||
|
.expect("acked");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The newest rows come back newest first, no more than asked for, and
|
||||||
|
/// the stream keeps 500 of them.
|
||||||
|
#[tokio::test]
|
||||||
|
async fn the_deliveries_are_the_newest_rows_and_the_log_keeps_500() {
|
||||||
|
let nats = TestNats::start();
|
||||||
|
let client = nats.client().await;
|
||||||
|
let js = jetstream::new(client.clone());
|
||||||
|
log::open_or_create(&js).await.expect("the log stream");
|
||||||
|
tokio::spawn(serve(client.clone()));
|
||||||
|
for n in 1..=3 {
|
||||||
|
log::append(&js, &row(n)).await;
|
||||||
|
}
|
||||||
|
|
||||||
|
let all: Deliveries =
|
||||||
|
serde_json::from_slice(&ask(&client, DELIVERIES_SUBJECT, Vec::new()).await)
|
||||||
|
.expect("an answer");
|
||||||
|
let numbers: Vec<u64> = all.rows.iter().map(|r| r.number).collect();
|
||||||
|
assert_eq!(numbers, [3, 2, 1]);
|
||||||
|
let first = Row {
|
||||||
|
at: all.rows[0].at,
|
||||||
|
..row(3)
|
||||||
|
};
|
||||||
|
assert_eq!(all.rows[0], first);
|
||||||
|
|
||||||
|
let two: Deliveries = serde_json::from_slice(
|
||||||
|
&ask(&client, DELIVERIES_SUBJECT, br#"{"limit":2}"#.to_vec()).await,
|
||||||
|
)
|
||||||
|
.expect("an answer");
|
||||||
|
assert_eq!(
|
||||||
|
two.rows.iter().map(|r| r.number).collect::<Vec<_>>(),
|
||||||
|
[3, 2]
|
||||||
|
);
|
||||||
|
|
||||||
|
for n in 4..=503 {
|
||||||
|
log::append(&js, &row(n)).await;
|
||||||
|
}
|
||||||
|
let mut stream = js.get_stream(LOG_STREAM).await.expect("the stream");
|
||||||
|
assert_eq!(stream.info().await.expect("info").state.messages, 500);
|
||||||
|
let newest: Deliveries = serde_json::from_slice(
|
||||||
|
&ask(&client, DELIVERIES_SUBJECT, br#"{"limit":9999}"#.to_vec()).await,
|
||||||
|
)
|
||||||
|
.expect("an answer");
|
||||||
|
assert_eq!(newest.rows.len(), 500);
|
||||||
|
assert_eq!(newest.rows[0].number, 503);
|
||||||
|
assert_eq!(newest.rows[499].number, 4);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A dry run counts the newest 50 events, and delivers nothing: no todo
|
||||||
|
/// op, no log row.
|
||||||
|
#[tokio::test]
|
||||||
|
async fn a_dry_run_counts_the_newest_fifty_events_and_delivers_nothing() {
|
||||||
|
let nats = TestNats::start();
|
||||||
|
let client = nats.client().await;
|
||||||
|
let js = jetstream::new(client.clone());
|
||||||
|
swarm_queue_client::forge_events::open_or_create(&client)
|
||||||
|
.await
|
||||||
|
.expect("events");
|
||||||
|
swarm_queue_client::router_todo::open_or_create(&client)
|
||||||
|
.await
|
||||||
|
.expect("todos");
|
||||||
|
log::open_or_create(&js).await.expect("the log stream");
|
||||||
|
tokio::spawn(serve(client.clone()));
|
||||||
|
// 60 assign events on distinct issues, then one on a type no flow
|
||||||
|
// starts from.
|
||||||
|
for n in 1..=60 {
|
||||||
|
publish_event(&js, "issue_assign", n).await;
|
||||||
|
}
|
||||||
|
publish_event(&js, "issue_comment", 61).await;
|
||||||
|
|
||||||
|
let request = serde_json::to_vec(&DryRunRequest {
|
||||||
|
document: Document::default(),
|
||||||
|
})
|
||||||
|
.unwrap();
|
||||||
|
let reply = ask(&client, DRY_RUN_SUBJECT, request).await;
|
||||||
|
let done: DryRun = serde_json::from_slice(&reply).expect("a dry run");
|
||||||
|
assert_eq!(done.events, 50);
|
||||||
|
let hits = |flow: &str, node: &str| {
|
||||||
|
done.flows
|
||||||
|
.iter()
|
||||||
|
.find(|f| f.flow == flow)
|
||||||
|
.and_then(|f| f.nodes.iter().find(|n| n.node == node))
|
||||||
|
.map(|n| n.hits)
|
||||||
|
};
|
||||||
|
// 49 of the newest 50 are assign events; the 50th is the comment.
|
||||||
|
assert_eq!(hits("assigned", "source"), Some(49));
|
||||||
|
assert_eq!(hits("assigned", "sink"), Some(49));
|
||||||
|
assert_eq!(hits("unassigned", "sink"), Some(0));
|
||||||
|
|
||||||
|
for (stream, what) in [(router_todo::STREAM, "todo op"), (LOG_STREAM, "log row")] {
|
||||||
|
let mut s = js.get_stream(stream).await.expect("the stream");
|
||||||
|
assert_eq!(s.info().await.expect("info").state.messages, 0, "a {what}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A document that fails its check is refused, and says why.
|
||||||
|
#[tokio::test]
|
||||||
|
async fn a_document_that_fails_its_check_is_refused() {
|
||||||
|
let nats = TestNats::start();
|
||||||
|
let client = nats.client().await;
|
||||||
|
tokio::spawn(serve(client.clone()));
|
||||||
|
let doc = Document {
|
||||||
|
disabled_builtins: vec!["no-such-flow".to_owned()],
|
||||||
|
..Document::default()
|
||||||
|
};
|
||||||
|
let request = serde_json::to_vec(&DryRunRequest { document: doc }).unwrap();
|
||||||
|
let refused: Refused =
|
||||||
|
serde_json::from_slice(&ask(&client, DRY_RUN_SUBJECT, request).await)
|
||||||
|
.expect("a refusal");
|
||||||
|
assert!(refused.error.contains("no-such-flow"), "{}", refused.error);
|
||||||
|
let garbled = ask(&client, DRY_RUN_SUBJECT, b"nonsense".to_vec()).await;
|
||||||
|
assert!(serde_json::from_slice::<Refused>(&garbled).is_ok());
|
||||||
|
}
|
||||||
|
}
|
||||||
183
swarm-notify-router/src/dry_run.rs
Normal file
183
swarm-notify-router/src/dry_run.rs
Normal file
|
|
@ -0,0 +1,183 @@
|
||||||
|
//! Running a flows document over past events without delivering anything.
|
||||||
|
//!
|
||||||
|
//! Each event goes through the walk the router delivers with ([`flow::reached`]
|
||||||
|
//! and [`route::route`]), counting the nodes it reaches. The per-issue
|
||||||
|
//! snapshots start empty and carry from one event to the next, as the router's
|
||||||
|
//! do, so an assign event's `assignees_added` is the difference to the
|
||||||
|
//! previous replayed one. Every recipient counts as an agent.
|
||||||
|
|
||||||
|
use std::collections::BTreeMap;
|
||||||
|
|
||||||
|
use swarm_wire::notify_flows::Flow;
|
||||||
|
use swarm_wire::router_api::{DryRun, FlowHits, NodeHits};
|
||||||
|
|
||||||
|
use crate::event::{Delivery, Snapshot, fields};
|
||||||
|
use crate::{flow, route};
|
||||||
|
|
||||||
|
/// `flows` over `events`, oldest first, each a forge event type and body.
|
||||||
|
/// An event that names no issue or PR is not counted.
|
||||||
|
pub fn run(flows: &[Flow], events: &[(String, Vec<u8>)]) -> DryRun {
|
||||||
|
let mut hits: Vec<BTreeMap<&str, u64>> = flows
|
||||||
|
.iter()
|
||||||
|
.map(|f| f.nodes.iter().map(|n| (n.id.as_str(), 0)).collect())
|
||||||
|
.collect();
|
||||||
|
let mut snapshots: BTreeMap<String, Snapshot> = BTreeMap::new();
|
||||||
|
let mut run = 0;
|
||||||
|
for (event_type, body) in events {
|
||||||
|
let Some(delivery) = Delivery::parse(body) else {
|
||||||
|
continue;
|
||||||
|
};
|
||||||
|
run += 1;
|
||||||
|
let key = delivery.state_key();
|
||||||
|
let before = snapshots.remove(&key).unwrap_or_default();
|
||||||
|
let event_fields = fields(event_type, &delivery, &before);
|
||||||
|
for (flow, counts) in flows.iter().zip(&mut hits) {
|
||||||
|
for id in flow::reached(flow, event_type, &event_fields) {
|
||||||
|
if let Some(count) = counts.get_mut(id) {
|
||||||
|
*count += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let routed = route::route(event_type, &delivery, &before, flows, |_| true);
|
||||||
|
snapshots.insert(key, routed.next);
|
||||||
|
}
|
||||||
|
DryRun {
|
||||||
|
events: run,
|
||||||
|
flows: flows
|
||||||
|
.iter()
|
||||||
|
.zip(hits)
|
||||||
|
.map(|(flow, counts)| FlowHits {
|
||||||
|
flow: flow.id.clone(),
|
||||||
|
nodes: flow
|
||||||
|
.nodes
|
||||||
|
.iter()
|
||||||
|
.map(|n| NodeHits {
|
||||||
|
node: n.id.clone(),
|
||||||
|
hits: counts.get(n.id.as_str()).copied().unwrap_or(0),
|
||||||
|
})
|
||||||
|
.collect(),
|
||||||
|
})
|
||||||
|
.collect(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use swarm_wire::notify_flows::{Document, builtins};
|
||||||
|
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
fn body(action: &str, number: i64, assignees: &[&str]) -> Vec<u8> {
|
||||||
|
let people: Vec<_> = assignees
|
||||||
|
.iter()
|
||||||
|
.map(|a| serde_json::json!({ "login": a }))
|
||||||
|
.collect();
|
||||||
|
serde_json::to_vec(&serde_json::json!({
|
||||||
|
"action": action,
|
||||||
|
"repository": { "full_name": "o/r", "id": 1 },
|
||||||
|
"issue": { "number": number, "title": "t", "html_url": "u", "assignees": people },
|
||||||
|
"sender": { "login": "mara" },
|
||||||
|
}))
|
||||||
|
.expect("encodes")
|
||||||
|
}
|
||||||
|
|
||||||
|
fn count(r: &DryRun, flow: &str, node: &str) -> u64 {
|
||||||
|
r.flows
|
||||||
|
.iter()
|
||||||
|
.find(|f| f.flow == flow)
|
||||||
|
.and_then(|f| f.nodes.iter().find(|n| n.node == node))
|
||||||
|
.unwrap_or_else(|| panic!("no node {flow}/{node}"))
|
||||||
|
.hits
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The built-ins over an assign, a second assign that adds nobody, an
|
||||||
|
/// unassign and a close: each node counts the events that reached it.
|
||||||
|
#[test]
|
||||||
|
fn each_node_counts_the_events_that_reached_it() {
|
||||||
|
let (flows, _) = Document::default().active();
|
||||||
|
let events = vec![
|
||||||
|
("issue_assign".to_owned(), body("assigned", 5, &["atlas"])),
|
||||||
|
("issue_assign".to_owned(), body("assigned", 5, &["atlas"])),
|
||||||
|
("issue_assign".to_owned(), body("unassigned", 5, &[])),
|
||||||
|
("issues".to_owned(), body("closed", 5, &[])),
|
||||||
|
("issues".to_owned(), b"not json".to_vec()),
|
||||||
|
];
|
||||||
|
let r = run(&flows, &events);
|
||||||
|
assert_eq!(r.events, 4);
|
||||||
|
// The source sees all three assign events, the match and the sink both
|
||||||
|
// assigned ones, though the second adds nobody.
|
||||||
|
assert_eq!(count(&r, "assigned", "source"), 3);
|
||||||
|
assert_eq!(count(&r, "assigned", "match"), 2);
|
||||||
|
assert_eq!(count(&r, "assigned", "sink"), 2);
|
||||||
|
// Only the unassign reaches the unassigned flow's sink.
|
||||||
|
assert_eq!(count(&r, "unassigned", "source"), 3);
|
||||||
|
assert_eq!(count(&r, "unassigned", "sink"), 1);
|
||||||
|
// The close is an `issues` event, which these flows start from only
|
||||||
|
// in `closed`.
|
||||||
|
assert_eq!(count(&r, "closed", "source"), 1);
|
||||||
|
assert_eq!(count(&r, "closed", "sink"), 1);
|
||||||
|
// No pull request push: the classifier flow counts zero.
|
||||||
|
assert_eq!(count(&r, "head-classifier", "source"), 0);
|
||||||
|
assert_eq!(count(&r, "head-classifier", "request"), 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A condition on a field that depends on the previous event of the
|
||||||
|
/// issue (`assignees_added`) holds for the first assign event and not for
|
||||||
|
/// an identical second one.
|
||||||
|
#[test]
|
||||||
|
fn the_issues_snapshot_carries_from_one_replayed_event_to_the_next() {
|
||||||
|
use swarm_wire::notify_flows::{Condition, Edge, Node, NodeKind};
|
||||||
|
|
||||||
|
let node = |id: &str, kind| Node {
|
||||||
|
id: id.to_owned(),
|
||||||
|
kind,
|
||||||
|
};
|
||||||
|
let edge = |from: &str, to: &str| Edge {
|
||||||
|
from: from.to_owned(),
|
||||||
|
to: to.to_owned(),
|
||||||
|
};
|
||||||
|
let flow = Flow {
|
||||||
|
id: "added".to_owned(),
|
||||||
|
name: "added".to_owned(),
|
||||||
|
enabled: true,
|
||||||
|
nodes: vec![
|
||||||
|
node(
|
||||||
|
"source",
|
||||||
|
NodeKind::Source {
|
||||||
|
events: vec!["issue_assign".to_owned()],
|
||||||
|
},
|
||||||
|
),
|
||||||
|
node(
|
||||||
|
"atlas",
|
||||||
|
NodeKind::Match {
|
||||||
|
when: vec![Condition {
|
||||||
|
field: "assignees_added".to_owned(),
|
||||||
|
equals: "atlas".to_owned(),
|
||||||
|
}],
|
||||||
|
},
|
||||||
|
),
|
||||||
|
],
|
||||||
|
edges: vec![edge("source", "atlas")],
|
||||||
|
};
|
||||||
|
let assign = ("issue_assign".to_owned(), body("assigned", 5, &["atlas"]));
|
||||||
|
let r = run(&[flow], &[assign.clone(), assign]);
|
||||||
|
assert_eq!(count(&r, "added", "source"), 2);
|
||||||
|
assert_eq!(count(&r, "added", "atlas"), 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Only the flows the document turns on are run.
|
||||||
|
#[test]
|
||||||
|
fn a_switched_off_built_in_is_not_listed() {
|
||||||
|
let doc = Document {
|
||||||
|
disabled_builtins: vec!["closed".to_owned()],
|
||||||
|
..Document::default()
|
||||||
|
};
|
||||||
|
let (flows, _) = doc.active();
|
||||||
|
let r = run(&flows, &[]);
|
||||||
|
assert_eq!(r.events, 0);
|
||||||
|
let listed: Vec<&str> = r.flows.iter().map(|f| f.flow.as_str()).collect();
|
||||||
|
assert!(!listed.contains(&"closed"));
|
||||||
|
assert!(listed.contains(&"assigned"));
|
||||||
|
assert_eq!(listed.len(), builtins().len() - 1);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -43,10 +43,10 @@ pub struct ReviewRequest {
|
||||||
pub reviewer: String,
|
pub reviewer: String,
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Walk `flow` for one event of type `event_type`, calling `leaf` on each
|
/// Walk `flow` for one event of type `event_type`, calling `visit` on each node
|
||||||
/// `todo` and `request_reviewer` node the event reaches. Assumes
|
/// the event reaches: the `source`, each `match` it passes, and the `todo` and
|
||||||
/// [`Flow::validate`] passed.
|
/// `request_reviewer` nodes below. Assumes [`Flow::validate`] passed.
|
||||||
fn walk<'f>(flow: &'f Flow, event_type: &str, fields: &Fields, mut leaf: impl FnMut(&'f Node)) {
|
fn walk<'f>(flow: &'f Flow, event_type: &str, fields: &Fields, mut visit: impl FnMut(&'f Node)) {
|
||||||
let Some(root) = flow
|
let Some(root) = flow
|
||||||
.nodes
|
.nodes
|
||||||
.iter()
|
.iter()
|
||||||
|
|
@ -58,17 +58,30 @@ fn walk<'f>(flow: &'f Flow, event_type: &str, fields: &Fields, mut leaf: impl Fn
|
||||||
while let Some(id) = stack.pop() {
|
while let Some(id) = stack.pop() {
|
||||||
let Some(node) = flow.node(id) else { continue };
|
let Some(node) = flow.node(id) else { continue };
|
||||||
match &node.kind {
|
match &node.kind {
|
||||||
NodeKind::Source { .. } => stack.extend(flow.children(id).into_iter().rev()),
|
NodeKind::Source { .. } => {
|
||||||
|
visit(node);
|
||||||
|
stack.extend(flow.children(id).into_iter().rev());
|
||||||
|
}
|
||||||
NodeKind::Match { when } => {
|
NodeKind::Match { when } => {
|
||||||
if when.iter().all(|c| holds(c, fields)) {
|
if when.iter().all(|c| holds(c, fields)) {
|
||||||
|
visit(node);
|
||||||
stack.extend(flow.children(id).into_iter().rev());
|
stack.extend(flow.children(id).into_iter().rev());
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
NodeKind::Todo(_) | NodeKind::RequestReviewer { .. } => leaf(node),
|
NodeKind::Todo(_) | NodeKind::RequestReviewer { .. } => visit(node),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The ids of the nodes of `flow` that one event of type `event_type` reaches,
|
||||||
|
/// a `match` node counting only when it passes. Emits nothing: this is the
|
||||||
|
/// dry run's view of the walk that [`evaluate`] and [`requests`] make.
|
||||||
|
pub fn reached<'f>(flow: &'f Flow, event_type: &str, fields: &Fields) -> Vec<&'f str> {
|
||||||
|
let mut ids = Vec::new();
|
||||||
|
walk(flow, event_type, fields, |node| ids.push(node.id.as_str()));
|
||||||
|
ids
|
||||||
|
}
|
||||||
|
|
||||||
/// Walk `flow` for one event of type `event_type` and return the todo ops it
|
/// Walk `flow` for one event of type `event_type` and return the todo ops it
|
||||||
/// emits. Assumes [`Flow::validate`] passed.
|
/// emits. Assumes [`Flow::validate`] passed.
|
||||||
pub fn evaluate(flow: &Flow, event_type: &str, fields: &Fields) -> Vec<Action> {
|
pub fn evaluate(flow: &Flow, event_type: &str, fields: &Fields) -> Vec<Action> {
|
||||||
|
|
|
||||||
134
swarm-notify-router/src/log.rs
Normal file
134
swarm-notify-router/src/log.rs
Normal file
|
|
@ -0,0 +1,134 @@
|
||||||
|
//! The deliveries log: one row on the `NOTIFY_ROUTER_LOG` stream for each
|
||||||
|
//! delivery the router makes (`swarm_wire::router_api`), and reading the
|
||||||
|
//! newest rows back.
|
||||||
|
//!
|
||||||
|
//! The stream keeps the newest 500 rows. Appending is best effort: a log that
|
||||||
|
//! cannot be written is reported and never holds up a delivery.
|
||||||
|
|
||||||
|
use std::time::{SystemTime, UNIX_EPOCH};
|
||||||
|
|
||||||
|
use anyhow::{Context as _, Result};
|
||||||
|
use async_nats::jetstream::stream::{Config, DiscardPolicy, Stream};
|
||||||
|
use async_nats::jetstream::{self};
|
||||||
|
use futures_util::{StreamExt as _, stream};
|
||||||
|
use swarm_wire::router_api::{Kind, LOG_MAX_MSGS, LOG_STREAM, LOG_SUBJECT, Row};
|
||||||
|
|
||||||
|
/// How many messages are fetched at once.
|
||||||
|
const FETCH_AT_ONCE: usize = 16;
|
||||||
|
|
||||||
|
/// Open the stream, creating it if nothing has yet. The router is its one
|
||||||
|
/// writer and its one creator.
|
||||||
|
pub async fn open_or_create(js: &jetstream::Context) -> Result<Stream> {
|
||||||
|
match js.get_stream(LOG_STREAM).await {
|
||||||
|
Ok(stream) => Ok(stream),
|
||||||
|
Err(e) => {
|
||||||
|
tracing::info!(stream = LOG_STREAM, reason = %e, "notify router: log stream not available, creating it");
|
||||||
|
js.create_stream(Config {
|
||||||
|
name: LOG_STREAM.to_owned(),
|
||||||
|
description: Some("Notification router deliveries, newest 500".to_owned()),
|
||||||
|
subjects: vec![LOG_SUBJECT.to_owned()],
|
||||||
|
max_messages: LOG_MAX_MSGS,
|
||||||
|
discard: DiscardPolicy::Old,
|
||||||
|
..Default::default()
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
.context("creating the log stream")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The event a delivery answers.
|
||||||
|
#[derive(Clone, Debug)]
|
||||||
|
pub struct Cause {
|
||||||
|
pub event: String,
|
||||||
|
pub repo: String,
|
||||||
|
pub number: u64,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Cause {
|
||||||
|
/// A row of `kind` made at `flow`'s `node` for `target`, which went
|
||||||
|
/// through when `result` is `Ok`.
|
||||||
|
pub fn row(
|
||||||
|
&self,
|
||||||
|
flow: &str,
|
||||||
|
node: &str,
|
||||||
|
kind: Kind,
|
||||||
|
target: &str,
|
||||||
|
result: Result<String, String>,
|
||||||
|
) -> Row {
|
||||||
|
let (ok, detail) = match result {
|
||||||
|
Ok(detail) => (true, detail),
|
||||||
|
Err(why) => (false, why),
|
||||||
|
};
|
||||||
|
Row {
|
||||||
|
at: SystemTime::now()
|
||||||
|
.duration_since(UNIX_EPOCH)
|
||||||
|
.map_or(0, |d| d.as_secs()),
|
||||||
|
flow: flow.to_owned(),
|
||||||
|
node: node.to_owned(),
|
||||||
|
event: self.event.clone(),
|
||||||
|
repo: self.repo.clone(),
|
||||||
|
number: self.number,
|
||||||
|
kind,
|
||||||
|
target: target.to_owned(),
|
||||||
|
ok,
|
||||||
|
detail,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Append `row`; a failure is logged and dropped.
|
||||||
|
pub async fn append(js: &jetstream::Context, row: &Row) {
|
||||||
|
let body = serde_json::to_vec(row).expect("a row encodes");
|
||||||
|
let sent = async {
|
||||||
|
js.publish(LOG_SUBJECT, body.into())
|
||||||
|
.await
|
||||||
|
.context("publishing")?
|
||||||
|
.await
|
||||||
|
.context("waiting for the ack")
|
||||||
|
};
|
||||||
|
if let Err(e) = sent.await {
|
||||||
|
tracing::warn!(error = format!("{e:#}"), flow = %row.flow, "notify router: could not log a delivery");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The payloads of the newest `limit` messages of `stream`, newest first,
|
||||||
|
/// with their subjects. A sequence that no longer exists is skipped.
|
||||||
|
pub async fn newest(
|
||||||
|
js: &jetstream::Context,
|
||||||
|
stream: &str,
|
||||||
|
limit: usize,
|
||||||
|
) -> Result<Vec<(String, Vec<u8>)>> {
|
||||||
|
let stream = js
|
||||||
|
.get_stream(stream)
|
||||||
|
.await
|
||||||
|
.with_context(|| format!("opening {stream}"))?;
|
||||||
|
let state = &stream.cached_info().state;
|
||||||
|
let (first, last) = (state.first_sequence, state.last_sequence);
|
||||||
|
let count = u64::try_from(limit).unwrap_or(u64::MAX);
|
||||||
|
let from = first.max((last + 1).saturating_sub(count));
|
||||||
|
let seqs = (from..=last).rev();
|
||||||
|
let fetched: Vec<_> = stream::iter(seqs)
|
||||||
|
.map(|seq| {
|
||||||
|
let stream = &stream;
|
||||||
|
async move { stream.get_raw_message(seq).await }
|
||||||
|
})
|
||||||
|
.buffered(FETCH_AT_ONCE)
|
||||||
|
.collect()
|
||||||
|
.await;
|
||||||
|
Ok(fetched
|
||||||
|
.into_iter()
|
||||||
|
.filter_map(Result::ok)
|
||||||
|
.map(|m| (m.subject.to_string(), m.payload.to_vec()))
|
||||||
|
.collect())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The newest `limit` rows of the log, newest first. A row that does not
|
||||||
|
/// decode is skipped.
|
||||||
|
pub async fn recent(js: &jetstream::Context, limit: usize) -> Result<Vec<Row>> {
|
||||||
|
Ok(newest(js, LOG_STREAM, limit)
|
||||||
|
.await?
|
||||||
|
.into_iter()
|
||||||
|
.filter_map(|(_, body)| serde_json::from_slice(&body).ok())
|
||||||
|
.collect())
|
||||||
|
}
|
||||||
|
|
@ -9,10 +9,15 @@
|
||||||
//!
|
//!
|
||||||
//! A `request_reviewer` node is a request to swarm-controller, which holds the
|
//! A `request_reviewer` node is a request to swarm-controller, which holds the
|
||||||
//! forge token (`swarm_wire::router_sink`). It is sent once, after the event's
|
//! forge token (`swarm_wire::router_sink`). It is sent once, after the event's
|
||||||
//! ops are queued and its snapshot is written, and the answer is only logged:
|
//! ops are queued and its snapshot is written, and the answer is only logged and
|
||||||
|
//! recorded:
|
||||||
//! a controller that is down or slow must not hold up every agent's todos, and
|
//! a controller that is down or slow must not hold up every agent's todos, and
|
||||||
//! the next push asks again.
|
//! the next push asks again.
|
||||||
//!
|
//!
|
||||||
|
//! Each delivery, made or failed, is a row of the deliveries log ([`log`]), and
|
||||||
|
//! the router answers swarm-controller's requests for the newest rows and for a
|
||||||
|
//! dry run of a flows document ([`api`], [`dry_run`]).
|
||||||
|
//!
|
||||||
//! Per issue it keeps a snapshot in its state bucket ([`event::Snapshot`]):
|
//! Per issue it keeps a snapshot in its state bucket ([`event::Snapshot`]):
|
||||||
//! the assignees it last saw and the agents it queued a todo for. An event is
|
//! the assignees it last saw and the agents it queued a todo for. An event is
|
||||||
//! acked once its ops are queued and the snapshot is written, so a crash
|
//! acked once its ops are queued and the snapshot is written, so a crash
|
||||||
|
|
@ -32,14 +37,20 @@ use async_nats::jetstream::{self, consumer::pull, kv, message::PublishMessage};
|
||||||
use futures_util::{StreamExt as _, TryStreamExt as _};
|
use futures_util::{StreamExt as _, TryStreamExt as _};
|
||||||
use swarm_queue_client::{ClientSecret, QueueConfig};
|
use swarm_queue_client::{ClientSecret, QueueConfig};
|
||||||
use swarm_secret_client::stored_secret::StoredSecret;
|
use swarm_secret_client::stored_secret::StoredSecret;
|
||||||
|
use swarm_wire::router_api::Kind;
|
||||||
use swarm_wire::router_sink::{self, Outcome, RequestReviewer};
|
use swarm_wire::router_sink::{self, Outcome, RequestReviewer};
|
||||||
use swarm_wire::{forge_events, notify_router, router_todo};
|
use swarm_wire::{forge_events, notify_router, router_todo};
|
||||||
|
|
||||||
|
mod api;
|
||||||
|
mod dry_run;
|
||||||
mod event;
|
mod event;
|
||||||
mod flow;
|
mod flow;
|
||||||
|
mod log;
|
||||||
mod route;
|
mod route;
|
||||||
mod settle;
|
mod settle;
|
||||||
mod stored;
|
mod stored;
|
||||||
|
#[cfg(test)]
|
||||||
|
mod test_nats;
|
||||||
|
|
||||||
/// The cert-auth role this service logs in to the store as. `swarm-bao.nix`
|
/// The cert-auth role this service logs in to the store as. `swarm-bao.nix`
|
||||||
/// writes it, matching the leaf's subject.
|
/// writes it, matching the leaf's subject.
|
||||||
|
|
@ -93,6 +104,7 @@ async fn main() -> Result<()> {
|
||||||
.await?;
|
.await?;
|
||||||
|
|
||||||
let mut router = Router::open(&queue).await?;
|
let mut router = Router::open(&queue).await?;
|
||||||
|
tokio::spawn(api::serve(queue.clone()));
|
||||||
tracing::info!(
|
tracing::info!(
|
||||||
stream = forge_events::STREAM,
|
stream = forge_events::STREAM,
|
||||||
"notify router: routing forge events"
|
"notify router: routing forge events"
|
||||||
|
|
@ -126,6 +138,9 @@ impl Router {
|
||||||
swarm_queue_client::router_todo::open_or_create(queue)
|
swarm_queue_client::router_todo::open_or_create(queue)
|
||||||
.await
|
.await
|
||||||
.context("opening the router todo stream")?;
|
.context("opening the router todo stream")?;
|
||||||
|
log::open_or_create(&js)
|
||||||
|
.await
|
||||||
|
.context("opening the deliveries log stream")?;
|
||||||
let events = events
|
let events = events
|
||||||
.create_consumer(pull::Config {
|
.create_consumer(pull::Config {
|
||||||
durable_name: Some(forge_events::ROUTER_CONSUMER.to_owned()),
|
durable_name: Some(forge_events::ROUTER_CONSUMER.to_owned()),
|
||||||
|
|
@ -199,8 +214,13 @@ impl Router {
|
||||||
let routed = route::route(&event_type, &delivery, &snapshot, &self.flows.active, |a| {
|
let routed = route::route(&event_type, &delivery, &snapshot, &self.flows.active, |a| {
|
||||||
roster.contains(a)
|
roster.contains(a)
|
||||||
});
|
});
|
||||||
|
let cause = log::Cause {
|
||||||
|
event: event_type.clone(),
|
||||||
|
repo: delivery.repo.clone(),
|
||||||
|
number: u64::try_from(delivery.number).unwrap_or_default(),
|
||||||
|
};
|
||||||
for action in &routed.actions {
|
for action in &routed.actions {
|
||||||
self.queue(seq, action).await.map_err(Skip::Retry)?;
|
self.queue(seq, &cause, action).await.map_err(Skip::Retry)?;
|
||||||
}
|
}
|
||||||
if !routed.skipped.is_empty() {
|
if !routed.skipped.is_empty() {
|
||||||
tracing::info!(%event_type, repo = %delivery.repo, number = delivery.number, skipped = ?routed.skipped, "notify router: recipients that are not agents got nothing");
|
tracing::info!(%event_type, repo = %delivery.repo, number = delivery.number, skipped = ?routed.skipped, "notify router: recipients that are not agents got nothing");
|
||||||
|
|
@ -210,14 +230,14 @@ impl Router {
|
||||||
self.state.put(&key, bytes.into()).await.map_err(retry)?;
|
self.state.put(&key, bytes.into()).await.map_err(retry)?;
|
||||||
}
|
}
|
||||||
for request in &routed.requests {
|
for request in &routed.requests {
|
||||||
self.request_review(request).await;
|
self.request_review(&cause, request).await;
|
||||||
}
|
}
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Ask swarm-controller for one review and log what it answered. One
|
/// Ask swarm-controller for one review, log what it answered and record
|
||||||
/// attempt: nothing here fails the event.
|
/// the delivery. One attempt: nothing here fails the event.
|
||||||
async fn request_review(&self, request: &flow::ReviewRequest) {
|
async fn request_review(&self, cause: &log::Cause, request: &flow::ReviewRequest) {
|
||||||
let wire = RequestReviewer {
|
let wire = RequestReviewer {
|
||||||
repo: request.repo.clone(),
|
repo: request.repo.clone(),
|
||||||
number: request.number,
|
number: request.number,
|
||||||
|
|
@ -233,7 +253,7 @@ impl Router {
|
||||||
request.number,
|
request.number,
|
||||||
&request.reviewer,
|
&request.reviewer,
|
||||||
);
|
);
|
||||||
match self
|
let result = match self
|
||||||
.nats
|
.nats
|
||||||
.send_request(router_sink::REQUEST_REVIEWER_SUBJECT, sent)
|
.send_request(router_sink::REQUEST_REVIEWER_SUBJECT, sent)
|
||||||
.await
|
.await
|
||||||
|
|
@ -241,22 +261,46 @@ impl Router {
|
||||||
Ok(reply) => match Outcome::decode(&String::from_utf8_lossy(&reply.payload)) {
|
Ok(reply) => match Outcome::decode(&String::from_utf8_lossy(&reply.payload)) {
|
||||||
Outcome::Requested => {
|
Outcome::Requested => {
|
||||||
tracing::info!(%flow, %repo, number, %reviewer, "notify router: review requested");
|
tracing::info!(%flow, %repo, number, %reviewer, "notify router: review requested");
|
||||||
|
Ok("requested".to_owned())
|
||||||
}
|
}
|
||||||
Outcome::Skipped(why) => {
|
Outcome::Skipped(why) => {
|
||||||
tracing::info!(%flow, %repo, number, %reviewer, %why, "notify router: review not requested");
|
tracing::info!(%flow, %repo, number, %reviewer, %why, "notify router: review not requested");
|
||||||
|
Ok(format!("skipped: {why}"))
|
||||||
}
|
}
|
||||||
Outcome::Error(why) => {
|
Outcome::Error(why) => {
|
||||||
tracing::warn!(%flow, %repo, number, %reviewer, %why, "notify router: the controller could not request the review");
|
tracing::warn!(%flow, %repo, number, %reviewer, %why, "notify router: the controller could not request the review");
|
||||||
|
Err(why)
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
Err(e) => {
|
Err(e) => {
|
||||||
tracing::warn!(%flow, %repo, number, %reviewer, error = %e, "notify router: no answer to a review request; not retried");
|
tracing::warn!(%flow, %repo, number, %reviewer, error = %e, "notify router: no answer to a review request; not retried");
|
||||||
|
Err(format!("no answer: {e}"))
|
||||||
}
|
}
|
||||||
}
|
};
|
||||||
|
let row = cause.row(flow, &request.node, Kind::RequestReviewer, reviewer, result);
|
||||||
|
log::append(&self.js, &row).await;
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Queue one op on its agent's subject and wait for the stream's ack.
|
/// Queue one op on its agent's subject, wait for the stream's ack and
|
||||||
async fn queue(&self, seq: u64, action: &flow::Action) -> Result<()> {
|
/// record the delivery, failed or not.
|
||||||
|
async fn queue(&self, seq: u64, cause: &log::Cause, action: &flow::Action) -> Result<()> {
|
||||||
|
let sent = self.publish(seq, action).await;
|
||||||
|
let result = match &sent {
|
||||||
|
Ok(()) => Ok(format!("{} {}", op_name(action.op.op), action.op.key)),
|
||||||
|
Err(e) => Err(format!("{e:#}")),
|
||||||
|
};
|
||||||
|
let row = cause.row(
|
||||||
|
&action.flow,
|
||||||
|
&action.node,
|
||||||
|
Kind::Todo,
|
||||||
|
&action.agent,
|
||||||
|
result,
|
||||||
|
);
|
||||||
|
log::append(&self.js, &row).await;
|
||||||
|
sent
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn publish(&self, seq: u64, action: &flow::Action) -> Result<()> {
|
||||||
let subject = router_todo::subject(&action.agent);
|
let subject = router_todo::subject(&action.agent);
|
||||||
let body = serde_json::to_vec(&action.op).expect("an op encodes");
|
let body = serde_json::to_vec(&action.op).expect("an op encodes");
|
||||||
let id = format!("{seq}:{}:{}:{}", action.flow, action.node, action.agent);
|
let id = format!("{seq}:{}:{}:{}", action.flow, action.node, action.agent);
|
||||||
|
|
@ -319,6 +363,14 @@ fn retry(e: impl std::error::Error + Send + Sync + 'static) -> Skip {
|
||||||
Skip::Retry(e.into())
|
Skip::Retry(e.into())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The word the log uses for an op.
|
||||||
|
const fn op_name(op: router_todo::Op) -> &'static str {
|
||||||
|
match op {
|
||||||
|
router_todo::Op::Upsert => "upsert",
|
||||||
|
router_todo::Op::Clear => "clear",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
async fn client_secret(client_id: &str) -> Result<String> {
|
async fn client_secret(client_id: &str) -> Result<String> {
|
||||||
use swarm_secret_client::builder::{ClientSecret, client_secret_path};
|
use swarm_secret_client::builder::{ClientSecret, client_secret_path};
|
||||||
|
|
||||||
|
|
@ -338,3 +390,148 @@ async fn client_secret(client_id: &str) -> Result<String> {
|
||||||
),
|
),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use swarm_wire::notify_flows::builtins;
|
||||||
|
use swarm_wire::router_api::Row;
|
||||||
|
|
||||||
|
use super::*;
|
||||||
|
use crate::test_nats::TestNats;
|
||||||
|
|
||||||
|
fn cause() -> log::Cause {
|
||||||
|
log::Cause {
|
||||||
|
event: "issue_assign".to_owned(),
|
||||||
|
repo: "o/r".to_owned(),
|
||||||
|
number: 9,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn action() -> flow::Action {
|
||||||
|
let flow = builtins().remove(0);
|
||||||
|
let (agent, op) = (
|
||||||
|
"atlas".to_owned(),
|
||||||
|
router_todo::RouterTodoOp {
|
||||||
|
op: router_todo::Op::Upsert,
|
||||||
|
key: "forge:o/r#9:assignment".to_owned(),
|
||||||
|
prefix: false,
|
||||||
|
summary: "s".to_owned(),
|
||||||
|
source: "router:assigned".to_owned(),
|
||||||
|
},
|
||||||
|
);
|
||||||
|
flow::Action {
|
||||||
|
flow: flow.id,
|
||||||
|
node: "sink".to_owned(),
|
||||||
|
agent,
|
||||||
|
op,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn review() -> flow::ReviewRequest {
|
||||||
|
flow::ReviewRequest {
|
||||||
|
flow: "head-classifier".to_owned(),
|
||||||
|
node: "request".to_owned(),
|
||||||
|
repo: "o/r".to_owned(),
|
||||||
|
number: 9,
|
||||||
|
reviewer: "review-classifier".to_owned(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn rows(router: &Router) -> Vec<Row> {
|
||||||
|
log::recent(&router.js, 10).await.expect("the log")
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A todo op that was queued is a row that went through; one the stream
|
||||||
|
/// would not take is a row with the error, and the delivery still fails.
|
||||||
|
#[tokio::test]
|
||||||
|
async fn a_queued_todo_is_a_row_and_a_failed_one_is_a_row_with_the_error() {
|
||||||
|
let nats = TestNats::start();
|
||||||
|
let client = nats.client().await;
|
||||||
|
let router = Router::open(&client).await.expect("opens");
|
||||||
|
|
||||||
|
router.queue(5, &cause(), &action()).await.expect("queued");
|
||||||
|
let [row] = &rows(&router).await[..] else {
|
||||||
|
panic!("one row")
|
||||||
|
};
|
||||||
|
assert!(row.ok, "{row:?}");
|
||||||
|
assert_eq!(
|
||||||
|
(row.flow.as_str(), row.node.as_str(), row.target.as_str()),
|
||||||
|
("assigned", "sink", "atlas")
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
(row.event.as_str(), row.repo.as_str(), row.number),
|
||||||
|
("issue_assign", "o/r", 9)
|
||||||
|
);
|
||||||
|
assert_eq!(row.kind, Kind::Todo);
|
||||||
|
assert_eq!(row.detail, "upsert forge:o/r#9:assignment");
|
||||||
|
|
||||||
|
router
|
||||||
|
.js
|
||||||
|
.delete_stream(router_todo::STREAM)
|
||||||
|
.await
|
||||||
|
.expect("deleted");
|
||||||
|
router
|
||||||
|
.queue(6, &cause(), &action())
|
||||||
|
.await
|
||||||
|
.expect_err("no stream takes it");
|
||||||
|
let rows = rows(&router).await;
|
||||||
|
assert_eq!(rows.len(), 2);
|
||||||
|
assert!(!rows[0].ok && !rows[0].detail.is_empty(), "{:?}", rows[0]);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Each answer the controller can give is a row: requested and skipped
|
||||||
|
/// went through, an error did not, and no answer at all did not either.
|
||||||
|
#[tokio::test]
|
||||||
|
async fn a_review_request_is_a_row_whatever_the_controller_answers() {
|
||||||
|
let nats = TestNats::start();
|
||||||
|
let client = nats.client().await;
|
||||||
|
let router = Router::open(&client).await.expect("opens");
|
||||||
|
|
||||||
|
// Nobody answers.
|
||||||
|
router.request_review(&cause(), &review()).await;
|
||||||
|
let first = rows(&router).await.remove(0);
|
||||||
|
assert!(
|
||||||
|
!first.ok && first.detail.starts_with("no answer"),
|
||||||
|
"{first:?}"
|
||||||
|
);
|
||||||
|
assert_eq!(first.kind, Kind::RequestReviewer);
|
||||||
|
assert_eq!(first.target, "review-classifier");
|
||||||
|
|
||||||
|
let mut hears = client
|
||||||
|
.subscribe(router_sink::REQUEST_REVIEWER_SUBJECT)
|
||||||
|
.await
|
||||||
|
.expect("subscribed");
|
||||||
|
let answering = client.clone();
|
||||||
|
tokio::spawn(async move {
|
||||||
|
for line in [
|
||||||
|
"requested",
|
||||||
|
"skipped:a review stands",
|
||||||
|
"error:the forge said no",
|
||||||
|
] {
|
||||||
|
let m = hears.next().await.expect("a request");
|
||||||
|
answering
|
||||||
|
.publish(m.reply.expect("a reply subject"), line.into())
|
||||||
|
.await
|
||||||
|
.expect("answered");
|
||||||
|
}
|
||||||
|
});
|
||||||
|
for _ in 0..3 {
|
||||||
|
router.request_review(&cause(), &review()).await;
|
||||||
|
}
|
||||||
|
let got: Vec<(bool, String)> = rows(&router)
|
||||||
|
.await
|
||||||
|
.into_iter()
|
||||||
|
.take(3)
|
||||||
|
.rev()
|
||||||
|
.map(|r| (r.ok, r.detail))
|
||||||
|
.collect();
|
||||||
|
assert_eq!(
|
||||||
|
got,
|
||||||
|
[
|
||||||
|
(true, "requested".to_owned()),
|
||||||
|
(true, "skipped: a review stands".to_owned()),
|
||||||
|
(false, "the forge said no".to_owned()),
|
||||||
|
]
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
|
||||||
77
swarm-notify-router/src/test_nats.rs
Normal file
77
swarm-notify-router/src/test_nats.rs
Normal file
|
|
@ -0,0 +1,77 @@
|
||||||
|
//! A throwaway `nats-server` with `JetStream`, for the tests whose subject is
|
||||||
|
//! the server's own behaviour: what a stream keeps and what a request gets.
|
||||||
|
//!
|
||||||
|
//! The same shape as `swarm-queue-client`'s: the binary comes from
|
||||||
|
//! [`NATS_SERVER_ENV`], which `nix/checks.nix` and `nix/devshell.nix` export,
|
||||||
|
//! and a test that needs it panics when it is unset rather than skipping.
|
||||||
|
|
||||||
|
use std::path::Path;
|
||||||
|
use std::process::{Child, Command, Stdio};
|
||||||
|
use std::time::Duration;
|
||||||
|
|
||||||
|
/// The variable naming the `nats-server` binary.
|
||||||
|
const NATS_SERVER_ENV: &str = "HIVE_NATS_SERVER";
|
||||||
|
|
||||||
|
/// The subdirectory holding nothing but the server's ports file.
|
||||||
|
const PORTS_DIR: &str = "ports";
|
||||||
|
|
||||||
|
/// A running server on a loopback port it bound itself. Dropping it kills
|
||||||
|
/// the server and removes its directory.
|
||||||
|
pub struct TestNats {
|
||||||
|
child: Child,
|
||||||
|
dir: tempfile::TempDir,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl TestNats {
|
||||||
|
pub fn start() -> Self {
|
||||||
|
let bin = std::env::var(NATS_SERVER_ENV).unwrap_or_else(|_| {
|
||||||
|
panic!("{NATS_SERVER_ENV} is exported by nix/checks.nix and nix/devshell.nix")
|
||||||
|
});
|
||||||
|
let dir = tempfile::Builder::new()
|
||||||
|
.prefix("test-nats-")
|
||||||
|
.tempdir()
|
||||||
|
.expect("a fresh directory for the server");
|
||||||
|
std::fs::create_dir(dir.path().join(PORTS_DIR)).expect("the ports file directory");
|
||||||
|
let child = Command::new(&bin)
|
||||||
|
.args(["-a", "127.0.0.1", "-p", "-1", "-js", "-sd"])
|
||||||
|
.arg(dir.path().join("store"))
|
||||||
|
.arg("--ports_file_dir")
|
||||||
|
.arg(dir.path().join(PORTS_DIR))
|
||||||
|
.stdout(Stdio::null())
|
||||||
|
.stderr(Stdio::null())
|
||||||
|
.spawn()
|
||||||
|
.unwrap_or_else(|e| panic!("starting {bin}: {e}"));
|
||||||
|
Self { child, dir }
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A client connected to the server, once it accepts connections.
|
||||||
|
pub async fn client(&self) -> async_nats::Client {
|
||||||
|
for _ in 0..100 {
|
||||||
|
if let Some(url) = announced_url(&self.dir.path().join(PORTS_DIR))
|
||||||
|
&& let Ok(client) = async_nats::connect(&url).await
|
||||||
|
{
|
||||||
|
return client;
|
||||||
|
}
|
||||||
|
tokio::time::sleep(Duration::from_millis(100)).await;
|
||||||
|
}
|
||||||
|
panic!(
|
||||||
|
"nats-server in {} never accepted a connection",
|
||||||
|
self.dir.path().display()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The client URL in the ports file the server writes once it listens.
|
||||||
|
fn announced_url(dir: &Path) -> Option<String> {
|
||||||
|
let ports = std::fs::read_dir(dir).ok()?.next()?.ok()?;
|
||||||
|
let ports: serde_json::Value =
|
||||||
|
serde_json::from_slice(&std::fs::read(ports.path()).ok()?).ok()?;
|
||||||
|
ports["nats"][0].as_str().map(str::to_owned)
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Drop for TestNats {
|
||||||
|
fn drop(&mut self) {
|
||||||
|
let _ = self.child.kill();
|
||||||
|
let _ = self.child.wait();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -21,6 +21,7 @@ pub mod notices;
|
||||||
pub mod notify_flows;
|
pub mod notify_flows;
|
||||||
pub mod notify_router;
|
pub mod notify_router;
|
||||||
pub mod review_classifier;
|
pub mod review_classifier;
|
||||||
|
pub mod router_api;
|
||||||
pub mod router_sink;
|
pub mod router_sink;
|
||||||
pub mod router_todo;
|
pub mod router_todo;
|
||||||
pub mod subagent_term;
|
pub mod subagent_term;
|
||||||
|
|
|
||||||
160
swarm-wire/src/router_api.rs
Normal file
160
swarm-wire/src/router_api.rs
Normal file
|
|
@ -0,0 +1,160 @@
|
||||||
|
//! What the notification router records and answers, for swarm-controller's
|
||||||
|
//! `GET /api/notify/deliveries` and `GET /api/notify/dry-run`.
|
||||||
|
//!
|
||||||
|
//! The router appends one [`Row`] to the [`LOG_STREAM`] stream for each
|
||||||
|
//! delivery it makes, and keeps the newest [`LOG_MAX_MSGS`]. The controller
|
||||||
|
//! asks it two things as NATS requests: the newest rows ([`DELIVERIES_SUBJECT`])
|
||||||
|
//! and, for a flows document, how many of the newest forge events reach each
|
||||||
|
//! node ([`DRY_RUN_SUBJECT`]), run without delivering anything.
|
||||||
|
//!
|
||||||
|
//! Three crates name these: the router records and answers, the controller
|
||||||
|
//! asks, and the auth-callout responder grants each its side. A router that
|
||||||
|
//! predates this answers neither, so the controller reports that.
|
||||||
|
|
||||||
|
use std::time::Duration;
|
||||||
|
|
||||||
|
use serde::{Deserialize, Serialize};
|
||||||
|
|
||||||
|
use crate::notify_flows::Document;
|
||||||
|
|
||||||
|
/// The stream holding the deliveries log.
|
||||||
|
pub const LOG_STREAM: &str = "NOTIFY_ROUTER_LOG";
|
||||||
|
|
||||||
|
/// The one subject [`LOG_STREAM`] captures. The router is its one publisher.
|
||||||
|
pub const LOG_SUBJECT: &str = "$SWARM.router.log";
|
||||||
|
|
||||||
|
/// How many rows the stream keeps; the oldest goes first.
|
||||||
|
pub const LOG_MAX_MSGS: i64 = 500;
|
||||||
|
|
||||||
|
/// The request for the newest [`Row`]s ([`DeliveriesRequest`]).
|
||||||
|
pub const DELIVERIES_SUBJECT: &str = "$SWARM.router.api.deliveries";
|
||||||
|
|
||||||
|
/// The request to run a document over the newest forge events
|
||||||
|
/// ([`DryRunRequest`]).
|
||||||
|
pub const DRY_RUN_SUBJECT: &str = "$SWARM.router.api.dry-run";
|
||||||
|
|
||||||
|
/// How many of the newest forge events a dry run covers.
|
||||||
|
pub const DRY_RUN_EVENTS: usize = 50;
|
||||||
|
|
||||||
|
/// The rows a request gets when it names no limit.
|
||||||
|
pub const DEFAULT_ROWS: usize = 100;
|
||||||
|
|
||||||
|
/// How long the controller waits for the router. The router's reply grant
|
||||||
|
/// lasts this long.
|
||||||
|
pub const REPLY_DEADLINE: Duration = Duration::from_secs(10);
|
||||||
|
|
||||||
|
/// What a delivery was.
|
||||||
|
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||||
|
#[cfg_attr(feature = "schema", derive(utoipa::ToSchema))]
|
||||||
|
#[serde(rename_all = "snake_case")]
|
||||||
|
pub enum Kind {
|
||||||
|
/// A todo op queued for an agent.
|
||||||
|
Todo,
|
||||||
|
/// A request to the controller for a review.
|
||||||
|
RequestReviewer,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// One delivery the router made.
|
||||||
|
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||||
|
#[cfg_attr(feature = "schema", derive(utoipa::ToSchema))]
|
||||||
|
pub struct Row {
|
||||||
|
/// Unix seconds.
|
||||||
|
pub at: u64,
|
||||||
|
pub flow: String,
|
||||||
|
pub node: String,
|
||||||
|
/// The forge event type, such as `issue_assign`.
|
||||||
|
pub event: String,
|
||||||
|
/// `owner/name` of the repository the event concerns.
|
||||||
|
pub repo: String,
|
||||||
|
/// The issue or pull request number.
|
||||||
|
pub number: u64,
|
||||||
|
pub kind: Kind,
|
||||||
|
/// The agent that got the todo, or the reviewer asked for.
|
||||||
|
pub target: String,
|
||||||
|
/// Whether it went through. A review the controller skipped went through.
|
||||||
|
pub ok: bool,
|
||||||
|
/// What happened: the op and key, the controller's answer, or the error.
|
||||||
|
pub detail: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Ask for the newest rows, newest first.
|
||||||
|
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
|
||||||
|
pub struct DeliveriesRequest {
|
||||||
|
/// At most this many; [`DEFAULT_ROWS`] when absent. The log keeps
|
||||||
|
/// [`LOG_MAX_MSGS`], so no answer is longer.
|
||||||
|
#[serde(default)]
|
||||||
|
pub limit: Option<usize>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The router's answer to a [`DeliveriesRequest`].
|
||||||
|
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||||
|
#[cfg_attr(feature = "schema", derive(utoipa::ToSchema))]
|
||||||
|
pub struct Deliveries {
|
||||||
|
/// Newest first.
|
||||||
|
pub rows: Vec<Row>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Run `document` over the newest [`DRY_RUN_EVENTS`] forge events.
|
||||||
|
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||||
|
pub struct DryRunRequest {
|
||||||
|
pub document: Document,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// How many events reached one node.
|
||||||
|
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||||
|
#[cfg_attr(feature = "schema", derive(utoipa::ToSchema))]
|
||||||
|
pub struct NodeHits {
|
||||||
|
pub node: String,
|
||||||
|
/// Events that reached the node. A `match` node counts those that passed
|
||||||
|
/// it, a `source` node those it starts from.
|
||||||
|
pub hits: u64,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The hits of one flow that would run.
|
||||||
|
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||||
|
#[cfg_attr(feature = "schema", derive(utoipa::ToSchema))]
|
||||||
|
pub struct FlowHits {
|
||||||
|
pub flow: String,
|
||||||
|
/// Every node of the flow, those no event reached with `0`.
|
||||||
|
pub nodes: Vec<NodeHits>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The router's answer to a [`DryRunRequest`] it ran.
|
||||||
|
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||||
|
#[cfg_attr(feature = "schema", derive(utoipa::ToSchema))]
|
||||||
|
pub struct DryRun {
|
||||||
|
/// How many events were run, at most [`DRY_RUN_EVENTS`]: fewer when the
|
||||||
|
/// stream holds fewer.
|
||||||
|
pub events: u64,
|
||||||
|
/// The flows the document turns on: the built-ins it leaves on and its
|
||||||
|
/// custom flows that are enabled.
|
||||||
|
pub flows: Vec<FlowHits>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The router's answer to a request it could not serve.
|
||||||
|
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||||
|
pub struct Refused {
|
||||||
|
pub error: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
/// The log keeps the rows the deliveries view can ask for.
|
||||||
|
#[test]
|
||||||
|
fn a_request_never_asks_for_more_than_the_stream_keeps() {
|
||||||
|
assert!(i64::try_from(DEFAULT_ROWS).expect("small") <= LOG_MAX_MSGS);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_request_without_a_limit_decodes() {
|
||||||
|
let r: DeliveriesRequest = serde_json::from_str("{}").expect("empty request");
|
||||||
|
assert_eq!(r.limit, None);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn the_reply_deadline_leaves_the_controller_time_to_answer() {
|
||||||
|
assert!(REPLY_DEADLINE < Duration::from_secs(30));
|
||||||
|
}
|
||||||
|
}
|
||||||
Loading…
Reference in a new issue