Watch
0
0
Fork
You've already forked hyperhive
0

Compare commits

...
21 changed files with 2193 additions and 368 deletions

View file

@ -64,21 +64,60 @@ The router holds no forge token and no webhook secret, and registers no hook.
A flow is a graph of nodes joined by edges: one `source` node naming the forge
event types it starts from, `match` nodes that pass when every condition
holds, and `sink` nodes that emit one todo op per recipient. A match node can
holds, and `todo` nodes that emit one todo op per recipient. A match node can
have more than one child, and each one runs, so one event can reach several
sinks. A flow must be a tree: one source, one parent for every other node,
every node reachable, no children under a sink.
todos. A flow must be a tree: one source, one parent for every other node,
every node reachable, no children under a `todo`. The most is 64 nodes, 16
levels deep, and 100 custom flows. A `todo` node sends to the agents an event
field holds (`assignees_added`, `todo_holders`, …) or to one named agent.
The router ships the three flows above, each a source → match on `action` →
todo sink.
todo. The code defines them (`swarm_wire::notify_flows::builtins`), so a
build that changes one reaches every swarm. A custom flow has the same type and
the same evaluator, so it can do anything a built-in does.
**The stored document.** swarm-controller keeps the rest in one document, the
`all` key of the `notify-flows` bucket: the ids of the built-in flows you
switched off, and the custom flows. Only the controller writes it, through
`GET` and `PUT /api/notify/flows`. A `PUT` replaces the whole document in one
compare-and-set. The controller answers 400 unless every flow passes the tree
check above and reads only what exists: a source event from `issues`,
`issue_assign`, `pull_request` or `pull_request_assign`, and condition and
recipient fields from the event's field list. The same goes for the `{field}`
placeholders in a todo's key and text. A `PUT` must carry the `revision` it read, `0` for no
document, and gets 409 when the stored revision is another. Swarm UI's
[`/notifications`](ui.md#notification-flows) page shows the document and
switches built-ins.
**What the router runs.** It reads the key at most every 10 seconds, when an
event arrives, and runs the built-ins you left on, then the custom flows that
are on and pass the tree check. It logs a custom flow that fails the check,
leaves it out, and runs the rest.
| the key | the router runs |
| -------------------------------------- | ----------------------- |
| absent, deleted, or its bucket missing | the built-in flows |
| a document it reads | the above |
| a document it can't read | the flows it ran before |
| a read that fails | the flows it ran before |
"Can't read" means not JSON, not a flows document, or a `version` other than 1.
Before the router has read any document, "before" means the built-ins, so a bad
document never leaves it with no flows. The controller refuses to overwrite a
document of a version it doesn't know.
**Deploying in any order.** The controller, the router and the UI can be on
different hosts. A controller that predates this leaves the key absent, and the
router runs the built-ins as before. A router that predates this ignores the
document, so the page and the API show a state the router isn't yet running.
## Credentials and grants
| Principal | Grant |
| --------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `swarm-notify-router`, in its container | store: `read` on `swarm/services/<router client id>/oidc/client`. Queue: its own consumer on `SWARM_EVENTS` (create, pull, ack), publish on `$SWARM.router.todo.*`, create `ROUTER_TODOS`, its own `notify-router-state` bucket, and `STREAM.INFO` on the agent-status bucket, which lists the agents |
| swarm-controller | publish on `$SWARM.events.forge.*` |
| an agent, under its own credential | create its own consumer `todo-<agent>` in the form whose filter is `$SWARM.router.todo.<agent>`, pull from it, and ack what it delivered |
| Principal | Grant |
| --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `swarm-notify-router`, in its container | store: `read` on `swarm/services/<router client id>/oidc/client`. Queue: its own consumer on `SWARM_EVENTS` (create, pull, ack), publish on `$SWARM.router.todo.*`, create `ROUTER_TODOS`, its own `notify-router-state` bucket, read on the `notify-flows` bucket, and `STREAM.INFO` on the agent-status bucket, which lists the agents |
| swarm-controller | publish on `$SWARM.events.forge.*`; creates and writes the `notify-flows` bucket (every bucket, by its `$KV.*.>` grant) |
| an agent, under its own credential | create its own consumer `todo-<agent>` in the form whose filter is `$SWARM.router.todo.<agent>`, pull from it, and ack what it delivered |
No other client may publish on `$SWARM.router.todo.*` or
`$SWARM.events.forge.*`, so each has one writer. The agent's create subject

View file

@ -19,6 +19,7 @@ domain, covers host-level detail for one hive.
| `/schedules` | the scheduled prompts: add, edit, pause, resume and remove |
| `/broadcast` | send one message to several agents at once, picked from a grid of their icons |
| `/messages` | the swarm-wide message feed: every message between agents, and to them from the operator, reminders and compaction, with a history window and a filter on sender or recipient |
| `/notifications` | the notification router's flows: a switch for each built-in, and every flow drawn as a tree |
| `/issues` | a cross-repo issue report |
| `/review` | the review classifier: its recent runs, a try-out on a real PR, and its settings |
| `/builds` | the swarm builder's recent builds of agent config PRs, with what nix printed when one failed |
@ -216,6 +217,22 @@ Only an operator can change a schedule here or with `swarmctl schedule`; an
agent has no way to add one. The page needs the controller's `/api/schedules`
routes.
### Notification flows
`/notifications` lists the [notification router's flows](notify-router.md#flows)
in two groups. Each **built-in** flow has an on/off switch; switching one off
stops the router running it from its next event, and switching it on again
restores it. Each **custom** flow shows its name and whether it's on. Every
flow draws as an indented tree: the forge events it starts from, the
conditions under it, and the todo each branch ends in.
The page edits nothing else. Add custom flows with `PUT /api/notify/flows`.
A switch sends the whole flows document with the revision the page read, `0` if
it read no document. The controller refuses the switch when the stored revision
is another, including when someone wrote a document after the page read none, and
the page shows why. Reload to switch against the current document.
### Sending a message to an agent
Under an agent's terminal, in the preview and in its own tab, is a **message**

View file

@ -8,6 +8,7 @@ import { AlertsPage } from "./pages/AlertsPage.js";
import { BuildsPage } from "./pages/builds/BuildsPage.js";
import { BroadcastPage } from "./pages/broadcast/BroadcastPage.js";
import { MessagesPage } from "./pages/messages/MessagesPage.js";
import { NotificationsPage } from "./pages/notifications/NotificationsPage.js";
import { ComponentsPage } from "./pages/ComponentsPage.js";
import { JobsPage } from "./pages/JobsPage.js";
import { HivesPage } from "./pages/HivesPage.js";
@ -43,6 +44,7 @@ export function App() {
<Route path="/builds" component={BuildsPage} />
<Route path="/broadcast" component={BroadcastPage} />
<Route path="/messages" component={MessagesPage} />
<Route path="/notifications" component={NotificationsPage} />
<Route path="/issues" component={IssueReportPage} />
<Route path="/review" component={ReviewPage} />
<Route path="/components" component={ComponentsPage} />

View file

@ -0,0 +1,95 @@
// Wire types of swarm-controller's `GET/PUT /api/notify/flows` and the small
// pure helpers the notifications page draws a flow with.
export type Recipients = { field: string } | { agent: string };
export interface Condition {
field: string;
equals: string;
}
// A node as the API sends it: `kind` sits beside `id`.
export type FlowNode =
| { id: string; kind: "source"; events: string[] }
| { id: string; kind: "match"; when: Condition[] }
| {
id: string;
kind: "todo";
to: Recipients;
op: "upsert" | "clear";
key: string;
prefix: boolean;
text: string;
};
export interface Flow {
id: string;
name: string;
enabled: boolean;
nodes: FlowNode[];
edges: { from: string; to: string }[];
}
export interface FlowsView {
revision: number;
disabled_builtins: string[];
builtins: Flow[];
flows: Flow[];
}
// One node of a flow with its depth below the source, in the order a reader
// walks the tree: a parent, then each child's subtree in edge order.
export interface TreeRow {
node: FlowNode;
depth: number;
}
export function treeRows(flow: Flow): TreeRow[] {
const byId = new Map(flow.nodes.map((n) => [n.id, n]));
const root = flow.nodes.find((n) => n.kind === "source");
if (!root) return [];
const rows: TreeRow[] = [];
const seen = new Set<string>();
const walk = (id: string, depth: number) => {
const node = byId.get(id);
// The controller refuses a flow that is not a tree; a repeat here would
// mean a cycle, which `seen` stops from looping.
if (!node || seen.has(id)) return;
seen.add(id);
rows.push({ node, depth });
for (const e of flow.edges) if (e.from === id) walk(e.to, depth + 1);
};
walk(root.id, 0);
return rows;
}
export function describeRecipients(to: Recipients): string {
return "field" in to ? `{${to.field}}` : to.agent;
}
// One line saying what a node is and does.
export function describeNode(node: FlowNode): string {
switch (node.kind) {
case "source":
return `forge ${node.events.join(", ")}`;
case "match":
return node.when.length === 0
? "match anything"
: node.when.map((c) => `${c.field} = ${c.equals}`).join(" and ");
case "todo":
return node.op === "clear"
? `clear todos ${node.key}${node.prefix ? "*" : ""} of ${describeRecipients(node.to)}`
: `todo to ${describeRecipients(node.to)}, key ${node.key}: ${node.text}`;
}
}
// `disabled` with built-in `id` switched on or off, without repeats and in a
// stable order.
export function withBuiltin(
disabled: string[],
id: string,
enabled: boolean,
): string[] {
const rest = disabled.filter((d) => d !== id);
return (enabled ? rest : [...rest, id]).sort();
}

View file

@ -0,0 +1,46 @@
/* NotificationsPage — the flows drawn as indented trees. */
.notify-flows {
display: flex;
flex-direction: column;
gap: 0.75em;
}
.notify-flow-head {
display: flex;
flex-wrap: wrap;
align-items: center;
justify-content: space-between;
gap: 0.5em;
}
.notify-flow-name {
font-weight: 600;
}
.notify-flow-tree {
margin: 0.4em 0 0;
padding: 0;
list-style: none;
font-size: 0.9em;
}
.notify-node {
display: flex;
gap: 0.5em;
padding: 0.1em 0;
}
.notify-node-kind {
flex: none;
min-width: 3.5em;
color: var(--muted);
}
.notify-node-text {
overflow-wrap: anywhere;
}
.notify-off {
opacity: 0.6;
}

View file

@ -0,0 +1,144 @@
// <NotificationsPage> — the notification router's flows. Backed by
// swarm-controller's `GET/PUT /api/notify/flows`; the controller is the only
// writer of the stored document and the router reads it every few seconds.
//
// Read-only except for one switch per built-in flow. A flow draws as an
// indented tree, the shape the API validates it to. Custom flows are not
// editable here: they arrive through the API.
import { useState } from "preact/hooks";
import { ApiErrorPanel } from "@hive/shared/api-error-panel.js";
import { readApiError, type ProblemDetails } from "@hive/shared/api-error.js";
import { Card } from "../../ui/card/Card.js";
import { CheckboxField } from "../../ui/checkbox-field/CheckboxField.js";
import { Panel } from "../../ui/panel/Panel.js";
import {
RefreshIntervalPicker,
type RefreshIntervalMs,
} from "../../ui/refresh-interval/RefreshInterval.js";
import { fetchJson, usePolled } from "../../ui/polled/usePolled.js";
import {
describeNode,
treeRows,
withBuiltin,
type Flow,
type FlowsView,
} from "./FlowTypes.js";
import "./NotificationsPage.css";
const DEFAULT_INTERVAL_MS: RefreshIntervalMs = 30_000;
function FlowTree({ flow }: { flow: Flow }) {
return (
<ul class="notify-flow-tree">
{treeRows(flow).map(({ node, depth }) => (
<li
class="notify-node"
key={node.id}
style={{ paddingLeft: `${depth * 1.5}em` }}
>
<span class="notify-node-kind">{node.kind}</span>
<span class="notify-node-text">{describeNode(node)}</span>
</li>
))}
</ul>
);
}
export function NotificationsPage() {
const [intervalMs, setIntervalMs] =
useState<RefreshIntervalMs>(DEFAULT_INTERVAL_MS);
const {
data: view,
error,
refresh,
} = usePolled(intervalMs, (signal) =>
fetchJson<FlowsView>("/api/notify/flows", signal),
);
const [pending, setPending] = useState(false);
const [actionError, setActionError] = useState<ProblemDetails | null>(null);
// Switch one built-in on or off: the whole document goes back with the
// revision this page read (0 for none), so a change made elsewhere since is refused
// rather than overwritten.
async function setBuiltin(id: string, enabled: boolean) {
if (!view) return;
setPending(true);
setActionError(null);
try {
const r = await fetch("/api/notify/flows", {
method: "PUT",
headers: { "content-type": "application/json" },
body: JSON.stringify({
revision: view.revision,
disabled_builtins: withBuiltin(view.disabled_builtins, id, enabled),
flows: view.flows,
}),
});
if (!r.ok) setActionError(await readApiError(r));
} catch (err) {
setActionError({ detail: String(err) });
} finally {
setPending(false);
refresh();
}
}
return (
<Panel
title="notifications"
icon="🔔"
actions={
<RefreshIntervalPicker
id="notifications-refresh"
value={intervalMs}
onChange={setIntervalMs}
/>
}
>
{error ? (
<ApiErrorPanel context="failed to load the flows" problem={error} />
) : null}
{actionError ? (
<ApiErrorPanel
context="the flows were not changed"
problem={actionError}
/>
) : null}
{!error && view === null ? <p>loading…</p> : null}
{view ? (
<div class="notify-flows">
<h3>built-in flows</h3>
{view.builtins.map((flow) => {
const on = !view.disabled_builtins.includes(flow.id);
return (
<Card key={flow.id} class={on ? "" : "notify-off"}>
<div class="notify-flow-head">
<span class="notify-flow-name">{flow.name}</span>
<CheckboxField
id={`builtin-${flow.id}`}
label={on ? "on" : "off"}
checked={on}
disabled={pending}
onChange={(checked) => setBuiltin(flow.id, checked)}
/>
</div>
<FlowTree flow={flow} />
</Card>
);
})}
<h3>custom flows</h3>
{view.flows.length === 0 ? <p>no custom flows</p> : null}
{view.flows.map((flow) => (
<Card key={flow.id} class={flow.enabled ? "" : "notify-off"}>
<div class="notify-flow-head">
<span class="notify-flow-name">{flow.name}</span>
<span>{flow.enabled ? "on" : "off"}</span>
</div>
<FlowTree flow={flow} />
</Card>
))}
</div>
) : null}
</Panel>
);
}

View file

@ -56,6 +56,7 @@ const NAV_ITEMS: { href: string; label: string; accent: string }[] = [
{ href: "/schedules", label: "schedules", accent: "var(--cyan)" },
{ href: "/broadcast", label: "broadcast", accent: "var(--blue)" },
{ href: "/messages", label: "messages", accent: "var(--purple)" },
{ href: "/notifications", label: "notifications", accent: "var(--yellow)" },
{ href: "/issues", label: "issues", accent: "var(--yellow)" },
{ href: "/review", label: "review", accent: "var(--amber)" },
{ href: "/builds", label: "builds", accent: "var(--green)" },

View file

@ -0,0 +1,80 @@
// The notifications page's tree and wording helpers. Run by `npm test`.
import assert from "node:assert/strict";
import { test } from "node:test";
import {
describeNode,
treeRows,
withBuiltin,
} from "../src/pages/notifications/FlowTypes.ts";
const flow = {
id: "f",
name: "f",
enabled: true,
nodes: [
{ id: "s", kind: "source", events: ["issues", "issue_assign"] },
{ id: "m", kind: "match", when: [{ field: "action", equals: "closed" }] },
{ id: "m2", kind: "match", when: [] },
{
id: "t",
kind: "todo",
to: { field: "todo_holders" },
op: "clear",
key: "forge:{repo}#{number}:",
prefix: true,
text: "",
},
{
id: "u",
kind: "todo",
to: { agent: "triage" },
op: "upsert",
key: "k",
prefix: false,
text: "look",
},
],
edges: [
{ from: "s", to: "m" },
{ from: "m", to: "t" },
{ from: "s", to: "m2" },
{ from: "m2", to: "u" },
],
};
test("treeRows: each subtree follows its parent, with its depth", () => {
assert.deepEqual(
treeRows(flow).map((r) => [r.node.id, r.depth]),
[
["s", 0],
["m", 1],
["t", 2],
["m2", 1],
["u", 2],
],
);
});
test("treeRows: a cycle ends the walk and a flow with no source is empty", () => {
const cyclic = { ...flow, edges: [...flow.edges, { from: "t", to: "s" }] };
assert.equal(treeRows(cyclic).length, 5);
assert.deepEqual(treeRows({ ...flow, nodes: [] }), []);
});
test("describeNode: says what each kind does", () => {
assert.equal(describeNode(flow.nodes[0]), "forge issues, issue_assign");
assert.equal(describeNode(flow.nodes[1]), "action = closed");
assert.equal(describeNode(flow.nodes[2]), "match anything");
assert.equal(
describeNode(flow.nodes[3]),
"clear todos forge:{repo}#{number}:* of {todo_holders}",
);
assert.equal(describeNode(flow.nodes[4]), "todo to triage, key k: look");
});
test("withBuiltin: switching off adds once, switching on removes", () => {
assert.deepEqual(withBuiltin([], "b", false), ["b"]);
assert.deepEqual(withBuiltin(["b"], "b", false), ["b"]);
assert.deepEqual(withBuiltin(["c"], "a", false), ["a", "c"]);
assert.deepEqual(withBuiltin(["a", "b"], "a", true), ["b"]);
});

View file

@ -82,6 +82,11 @@ The swarm's control plane. The swarm UI and `swarmctl` are its clients.
`review-classifier-settings` KV bucket, which this daemon alone reads and
writes (`GET`/`PUT /api/review-classifier/settings`) →
[`docs/swarm/review-classifier.md`](../docs/swarm/review-classifier.md#swarm-level-settings).
- **Notification flows** — which of the notification router's built-in flows are
switched off, and the custom flows, as one document in the `notify-flows`
KV bucket, which this daemon alone writes (`GET`/`PUT /api/notify/flows`);
the router reads it →
[`docs/swarm/notify-router.md`](../docs/swarm/notify-router.md#flows).
- **Operator profiles** — each operator's name and the pronouns on their
forge profile, for every member of authelia's operator group, as one list
under the one key of the `operator-profiles` KV bucket. This daemon alone

View file

@ -2113,6 +2113,14 @@ mod tests {
events(DeliveryKind::PullRequest),
["pull_request", "pull_request_assign"]
);
// The events a stored flow's source may name are the ones the hooks
// deliver to the router.
let mut routed = events(DeliveryKind::Issues);
routed.extend(events(DeliveryKind::PullRequest));
routed.sort_unstable();
let mut sources = swarm_wire::notify_flows::SOURCE_EVENTS.to_vec();
sources.sort_unstable();
assert_eq!(routed, sources);
}
#[test]

View file

@ -50,6 +50,7 @@ mod jobq;
mod linked_accounts;
mod matrix_account;
mod message_stream;
mod notify_flows;
mod otel_http_client;
mod pulls;
mod queue_identity;

View file

@ -0,0 +1,446 @@
//! `GET` and `PUT /api/notify/flows`: which of the notification router's
//! built-in flows are switched off, and the custom flows.
//!
//! The flows are one document under one key ([`notify_flows::KEY`] in
//! [`notify_flows::BUCKET`]), which only this daemon writes, as the schedule
//! definitions are. A `PUT` replaces the whole document in one
//! compare-and-set, after [`Document::check`]. The router reads the key
//! every few seconds, so a change applies to the next event with no restart.
//! No document means the built-in flows alone.
//!
//! The built-in flows are defined in code, in `swarm_wire::notify_flows`, and
//! the document stores only their disabled ids, so a `GET` shows the ones
//! this build knows.
use anyhow::Context as _;
use async_nats::jetstream::kv::Store;
use axum::Json;
use axum::extract::State;
use axum::extract::rejection::JsonRejection;
use axum::http::StatusCode;
use problem_details::ProblemDetails;
use serde::{Deserialize, Serialize};
use swarm_wire::notify_flows::{self, DecodeError, Document, Flow};
use utoipa::ToSchema;
use crate::routes::{AppState, error_problem, wanted_error_status};
use crate::schedules::{queue, read_with, refused, write_revision};
/// How many times a losing writer re-reads and re-applies, as in
/// `wanted::MAX_ATTEMPTS`.
const MAX_ATTEMPTS: usize = 5;
/// What `GET` and a successful `PUT` answer.
#[derive(Debug, Clone, PartialEq, Serialize, ToSchema)]
pub(crate) struct FlowsView {
/// The revision of the stored document, to send back in a `PUT`; `0` when
/// none is stored.
revision: u64,
/// The ids of the built-in flows that are switched off.
disabled_builtins: Vec<String>,
/// The built-in flows this build ships, all of them, switched off or not.
builtins: Vec<Flow>,
/// The custom flows.
flows: Vec<Flow>,
}
/// The body of a `PUT`: the whole document.
#[derive(Debug, Clone, Deserialize, ToSchema)]
#[serde(deny_unknown_fields)]
pub(crate) struct PutFlows {
/// The `revision` the client read, `0` for "no document stored". The
/// `PUT` is refused when the stored revision is another, which includes a
/// document written since a client read `0`. A KV revision is never `0`.
revision: u64,
/// The ids of the built-in flows to switch off.
#[serde(default)]
disabled_builtins: Vec<String>,
/// The custom flows.
#[serde(default)]
flows: Vec<Flow>,
}
impl FlowsView {
fn of(doc: Document, revision: u64) -> Self {
Self {
revision,
disabled_builtins: doc.disabled_builtins,
builtins: notify_flows::builtins(),
flows: doc.flows,
}
}
}
/// What the key holds.
enum Stored {
/// No key, or a deleted one.
Nothing,
Document(Document),
/// A document this build cannot read, with why.
Unreadable(DecodeError),
}
/// The stored document, with the revision to write against.
async fn stored(store: &Store) -> anyhow::Result<(Stored, Option<u64>)> {
struct Read(Stored);
impl Default for Read {
fn default() -> Self {
Self(Stored::Nothing)
}
}
let (Read(stored), revision) = read_with(store, notify_flows::KEY, |bytes| {
Ok(Read(match Document::decode(bytes) {
Ok(doc) => Stored::Document(doc),
Err(e) => Stored::Unreadable(e),
}))
})
.await?;
Ok((stored, revision))
}
/// Open [`notify_flows::BUCKET`], creating it if nothing has yet.
async fn open(client: &async_nats::Client) -> anyhow::Result<Store> {
let js = async_nats::jetstream::new(client.clone());
if let Ok(store) = js.get_key_value(notify_flows::BUCKET).await {
return Ok(store);
}
js.create_key_value(async_nats::jetstream::kv::Config {
bucket: notify_flows::BUCKET.to_owned(),
description: "The notification router's flows: disabled built-ins and custom flows"
.to_owned(),
history: 1,
..Default::default()
})
.await
.with_context(|| format!("creating the {} bucket", notify_flows::BUCKET))
}
fn queue_problem(e: &anyhow::Error) -> ProblemDetails {
tracing::warn!(error = %format!("{e:#}"), "notify flows: queue failure");
error_problem(wanted_error_status(e), &format!("{e:#}"))
}
/// Read the flows. Creates the `notify-flows` bucket when it is missing, as a
/// `PUT` does, so the router finds it empty rather than absent.
#[utoipa::path(
get,
path = "/api/notify/flows",
responses(
(status = 200, description = "the stored document, with the built-in flows this \
build ships; with none stored, no disabled built-ins and no custom flows",
body = FlowsView),
(status = 409, description = "the stored document is not one this build reads, \
such as a newer build's version (problem+json)", body = String),
(status = 503, description = "no swarm queue is configured on this host, or it is \
not connected (problem+json)", body = String),
(status = 500, description = "the flows could not be read (problem+json)",
body = String),
),
tag = "notify"
)]
pub(crate) async fn get_flows(
State(state): State<AppState>,
) -> Result<Json<FlowsView>, ProblemDetails> {
let client = queue(&state).map_err(refused)?;
let store = open(&client).await.map_err(|e| queue_problem(&e))?;
let (stored, revision) = stored(&store).await.map_err(|e| queue_problem(&e))?;
match stored {
Stored::Nothing => Ok(Json(FlowsView::of(
Document::default(),
revision.unwrap_or(0),
))),
Stored::Document(doc) => Ok(Json(FlowsView::of(doc, revision.unwrap_or(0)))),
Stored::Unreadable(e) => Err(unreadable(&e)),
}
}
fn unreadable(why: &DecodeError) -> ProblemDetails {
error_problem(
StatusCode::CONFLICT,
&format!("the stored flows document is not one this build reads: {why}"),
)
}
/// Replace the flows.
#[utoipa::path(
put,
path = "/api/notify/flows",
request_body = PutFlows,
responses(
(status = 200, description = "the document as stored; the router applies it from \
its next event", body = FlowsView),
(status = 400, description = "not JSON or not this shape, or the document fails its \
check: a `disabled_builtins` entry that is no built-in, a custom flow whose id \
is another's or a built-in's, a flow that is not a tree (one `source`, one \
parent for every other node, every node reachable, no children under a \
`todo`), more than 64 nodes, 16 levels or 100 flows, an id that is not 1 to 64 \
letters, digits, `-` or `_`, a source event the router is not sent, a condition \
or recipient field that is no event field, or an upsert with no text \
(problem+json)", body = String),
(status = 409, description = "the stored document is not at `revision` (another write landed, or one now exists where the client read none), or the stored \
document is a version this build does not read, which a `PUT` would erase \
(problem+json)", body = String),
(status = 503, description = "no swarm queue is configured on this host, or it is \
not connected (problem+json)", body = String),
(status = 500, description = "the flows could not be written (problem+json)",
body = String),
),
tag = "notify"
)]
pub(crate) async fn put_flows(
State(state): State<AppState>,
body: Result<Json<PutFlows>, JsonRejection>,
) -> Result<Json<FlowsView>, ProblemDetails> {
let Json(put) = body.map_err(|e| error_problem(StatusCode::BAD_REQUEST, &e.body_text()))?;
let doc = Document {
version: notify_flows::VERSION,
disabled_builtins: put.disabled_builtins,
flows: put.flows,
};
doc.check()
.map_err(|why| error_problem(StatusCode::BAD_REQUEST, &why))?;
let client = queue(&state).map_err(refused)?;
let store = open(&client).await.map_err(|e| queue_problem(&e))?;
for _ in 0..MAX_ATTEMPTS {
let (current, revision) = stored(&store).await.map_err(|e| queue_problem(&e))?;
// An unreadable document is replaced only when it is not a version
// this build does not know: a newer build's document is kept for it.
if let Stored::Unreadable(why @ DecodeError::UnknownVersion(_)) = &current {
return Err(unreadable(why));
}
if put.revision != revision.unwrap_or(0) {
return Err(error_problem(
StatusCode::CONFLICT,
&format!(
"the flows are not at revision {}; read them again",
put.revision
),
));
}
if let Some(written) = write_revision(&store, notify_flows::KEY, &doc, revision)
.await
.map_err(|e| queue_problem(&e))?
{
tracing::info!(
disabled = ?doc.disabled_builtins,
custom = doc.flows.len(),
"set the notification router's flows"
);
return Ok(Json(FlowsView::of(doc, written)));
}
}
Err(queue_problem(&anyhow::anyhow!(
"gave up writing the notification flows after {MAX_ATTEMPTS} conflicting writes"
)))
}
#[cfg(test)]
mod tests {
use swarm_wire::notify_flows::{Edge, Node, NodeKind, Recipients, TodoSink};
use swarm_wire::router_todo::Op;
use super::*;
use crate::test_nats::TestNats;
fn custom(id: &str) -> Flow {
Flow {
id: id.to_owned(),
name: "new to triage".to_owned(),
enabled: true,
nodes: vec![
Node {
id: "source".to_owned(),
kind: NodeKind::Source {
events: vec!["issues".to_owned()],
},
},
Node {
id: "sink".to_owned(),
kind: NodeKind::Todo(TodoSink {
to: Recipients::Agent("triage".to_owned()),
op: Op::Upsert,
key: "forge:{repo}#{number}:triage".to_owned(),
prefix: false,
text: "triage {repo}#{number}".to_owned(),
}),
},
],
edges: vec![Edge {
from: "source".to_owned(),
to: "sink".to_owned(),
}],
}
}
fn body(revision: u64, disabled: &[&str], flows: Vec<Flow>) -> Json<PutFlows> {
Json(PutFlows {
revision,
disabled_builtins: disabled.iter().map(|s| (*s).to_owned()).collect(),
flows,
})
}
#[tokio::test]
async fn the_flows_are_read_replaced_and_checked_on_save() {
let nats = TestNats::start();
let client = nats.client().await;
let state = crate::agent_limits::tests::state(&client);
// Nothing stored: the built-ins alone, none off, no revision.
let Json(view) = get_flows(State(state.clone())).await.expect("get");
assert_eq!(view.revision, 0);
assert!(view.disabled_builtins.is_empty() && view.flows.is_empty());
assert_eq!(view.builtins.len(), 3);
let Json(saved) = put_flows(
State(state.clone()),
Ok(body(0, &["unassigned"], vec![custom("triage")])),
)
.await
.expect("put");
let revision = saved.revision;
assert_ne!(revision, 0);
let Json(read) = get_flows(State(state.clone())).await.expect("get");
assert_eq!(read, saved);
assert_eq!(read.disabled_builtins, ["unassigned"]);
assert_eq!(read.flows[0].id, "triage");
// The stored bytes are the document the router decodes.
let store = open(&client).await.expect("bucket");
let bytes = store
.get(notify_flows::KEY)
.await
.expect("read")
.expect("stored");
let doc = Document::decode(&bytes).expect("the router reads it");
assert_eq!(doc.active().0.len(), 3, "2 built-ins and the custom flow");
// A refused save changes nothing: an orphan node, and a built-in that does not exist.
let mut orphan = custom("orphan");
orphan.edges.clear();
let refused = put_flows(State(state.clone()), Ok(body(revision, &[], vec![orphan])))
.await
.expect_err("an orphan node");
assert_eq!(refused.status, Some(StatusCode::BAD_REQUEST), "{refused:?}");
let unknown = put_flows(State(state.clone()), Ok(body(revision, &["nope"], vec![])))
.await
.expect_err("no such built-in");
assert_eq!(unknown.status, Some(StatusCode::BAD_REQUEST), "{unknown:?}");
let Json(after) = get_flows(State(state.clone())).await.expect("get");
assert_eq!(after, saved);
// A stale revision is refused; the current one lands.
let outdated = put_flows(State(state.clone()), Ok(body(revision + 100, &[], vec![])))
.await
.expect_err("stale");
assert_eq!(outdated.status, Some(StatusCode::CONFLICT), "{outdated:?}");
let Json(next) = put_flows(State(state.clone()), Ok(body(revision, &[], vec![])))
.await
.expect("current revision");
assert!(next.revision > revision);
assert!(next.flows.is_empty());
}
/// A document of a version this build does not know is neither served
/// nor replaced, and a garbled one is replaced.
#[tokio::test]
async fn a_document_of_another_version_is_kept_and_a_garbled_one_is_replaced() {
let nats = TestNats::start();
let client = nats.client().await;
let state = crate::agent_limits::tests::state(&client);
let store = open(&client).await.expect("bucket");
store
.put(
notify_flows::KEY,
br#"{"version":2,"flows":[]}"#.to_vec().into(),
)
.await
.expect("seed");
let got = get_flows(State(state.clone()))
.await
.expect_err("version 2");
assert_eq!(got.status, Some(StatusCode::CONFLICT), "{got:?}");
let refused = put_flows(State(state.clone()), Ok(body(0, &[], vec![])))
.await
.expect_err("version 2 is kept");
assert_eq!(refused.status, Some(StatusCode::CONFLICT), "{refused:?}");
let kept = store
.get(notify_flows::KEY)
.await
.expect("read")
.expect("stored");
assert_eq!(&kept[..], br#"{"version":2,"flows":[]}"#);
// Control: garbage that names no version is replaced by a save.
let garbage = store
.put(notify_flows::KEY, b"garbage".to_vec().into())
.await
.expect("seed");
let _ = put_flows(State(state.clone()), Ok(body(garbage, &["closed"], vec![])))
.await
.expect("replaces garbage");
let Json(view) = get_flows(State(state)).await.expect("readable again");
assert_eq!(view.disabled_builtins, ["closed"]);
}
/// A page that read no document cannot erase one written since: its
/// `revision: 0` means "create", and the document exists by then.
#[tokio::test]
async fn a_page_that_read_nothing_cannot_erase_a_flow_written_after() {
let nats = TestNats::start();
let client = nats.client().await;
let state = crate::agent_limits::tests::state(&client);
let Json(page) = get_flows(State(state.clone())).await.expect("get");
assert_eq!(page.revision, 0);
// The API writes a custom flow while the page is open.
let Json(api) = put_flows(
State(state.clone()),
Ok(body(0, &[], vec![custom("triage")])),
)
.await
.expect("the API's write");
// The page's first switch carries what it read.
let erased = put_flows(
State(state.clone()),
Ok(body(page.revision, &["closed"], vec![])),
)
.await
.expect_err("the page read nothing, and a document exists");
assert_eq!(erased.status, Some(StatusCode::CONFLICT), "{erased:?}");
let Json(after) = get_flows(State(state.clone())).await.expect("get");
assert_eq!(after, api);
assert_eq!(after.flows[0].id, "triage");
// Control: re-read, the same switch lands and keeps the flow.
let Json(switched) = put_flows(
State(state),
Ok(body(after.revision, &["closed"], after.flows.clone())),
)
.await
.expect("the switch with the revision it read");
assert_eq!(switched.disabled_builtins, ["closed"]);
assert_eq!(switched.flows[0].id, "triage");
}
#[tokio::test]
async fn a_template_naming_no_event_field_is_refused_on_save() {
let nats = TestNats::start();
let client = nats.client().await;
let state = crate::agent_limits::tests::state(&client);
let mut typo = custom("typo");
if let NodeKind::Todo(sink) = &mut typo.nodes[1].kind {
sink.key = "forge:{repo}#{numbr}:triage".to_owned();
}
let refused = put_flows(State(state.clone()), Ok(body(0, &[], vec![typo])))
.await
.expect_err("{numbr} is no field");
assert_eq!(refused.status, Some(StatusCode::BAD_REQUEST), "{refused:?}");
// Control: the same flow spelled right is stored.
let _ = put_flows(State(state), Ok(body(0, &[], vec![custom("typo")])))
.await
.expect("{number} is a field");
}
}

View file

@ -31,8 +31,8 @@ use crate::{
agent_control, agent_icon, agent_input, agent_limits, agent_login, agent_permissions,
agent_renewal, agent_state_stream, agent_status, agents_md, alerts, archetypes, auth, builder,
claude_login, config_pr, forge, forge_account, github_account, inference, issue_report,
linked_accounts, matrix_account, message_stream, pulls, queue_identity, review_classifier,
schedules, status, subagent_term, term_stream, wanted, webhook,
linked_accounts, matrix_account, message_stream, notify_flows, pulls, queue_identity,
review_classifier, schedules, status, subagent_term, term_stream, wanted, webhook,
};
/// Where the daemon binds, overridable via `SWARM_CONTROLLER_SOCKET`.
@ -76,6 +76,7 @@ pub(crate) fn socket_path() -> PathBuf {
(name = "repos", description = "forge repo browsing (swarm-ui's issue-report page)"),
(name = "forge", description = "forge site admins (`swarmctl forge make-admin`)"),
(name = "schedules", description = "scheduled prompts (`swarmctl schedule`)"),
(name = "notify", description = "the notification router's flows: built-ins switched off, and custom flows"),
(name = "review-classifier", description = "trying review classifications against a PR (swarm-ui's try-out page), and the classifier's swarm-wide settings"),
(name = "pulls", description = "open pull requests, for the try-out page's dropdowns"),
(name = "alerts", description = "the swarm Grafana's current alerts (swarm-ui's alerts page)"),
@ -635,6 +636,7 @@ pub(crate) fn build_app(state: AppState) -> axum::Router {
review_classifier::settings::get_settings,
review_classifier::settings::put_settings
))
.routes(routes!(notify_flows::get_flows, notify_flows::put_flows))
.routes(routes!(webhook::post_webhook_forge))
.split_for_parts();
// Just the JSON, not the UI — Swagger UI itself is nginx-hosted from

View file

@ -317,7 +317,7 @@ impl Policy {
/// What the notification router may do: drain its own consumer on the
/// forge events stream, queue todo ops for any agent, keep its own state
/// bucket, and list which agents exist.
/// bucket, read the flows document, and list which agents exist.
///
/// - **Forge events.** Open or create the stream, create its one durable
/// in the extended form whose last tokens are the filter (the server
@ -330,14 +330,18 @@ impl Policy {
/// subjects, which is what makes the router their sole writer.
/// - **Its state bucket.** Create, look up, read and write
/// `notify-router-state`, and no other bucket.
/// - **The flows document.** `STREAM.INFO` and `DIRECT.GET` on
/// `notify-flows`, which it reads and never creates or writes: the
/// controller creates the bucket and is its one writer.
/// - **The roster.** `STREAM.INFO` on the agent-status bucket, which lists
/// its keys (`<hive>.<agent>`) and writes none.
fn notify_router_permissions(client_id: &str) -> Permissions {
use swarm_wire::{forge_events, notify_router, router_todo};
use swarm_wire::{forge_events, notify_flows, notify_router, router_todo};
let events = forge_events::STREAM;
let consumer = forge_events::ROUTER_CONSUMER;
let state = format!("KV_{}", notify_router::STATE_BUCKET);
let flows = format!("KV_{}", notify_flows::BUCKET);
let mut publish = Self::jetstream_minimum().to_vec();
publish.extend([
format!("$JS.API.STREAM.INFO.{events}"),
@ -355,6 +359,8 @@ impl Policy {
format!("$JS.API.STREAM.CREATE.{state}"),
format!("$JS.API.DIRECT.GET.{state}.>"),
format!("$KV.{}.>", notify_router::STATE_BUCKET),
format!("$JS.API.STREAM.INFO.{flows}"),
format!("$JS.API.DIRECT.GET.{flows}.>"),
format!("$JS.API.STREAM.INFO.{}", Self::agent_status_stream()),
Self::inbox(client_id),
]);
@ -1799,7 +1805,7 @@ mod tests {
#[test]
fn the_notify_router_drains_its_own_consumer_and_writes_only_its_own_state() {
use swarm_queue_client::client_inbox_prefix;
use swarm_wire::{forge_events, notify_router, router_todo};
use swarm_wire::{forge_events, notify_flows, notify_router, router_todo};
let p = with_router()
.permissions(NOTIFY_ROUTER)
@ -1807,6 +1813,7 @@ mod tests {
let may = |subject: &str| p.publish.iter().any(|g| covers(g, subject));
let events = forge_events::STREAM;
let state = notify_router::STATE_BUCKET;
let flows = notify_flows::BUCKET;
for allowed in [
format!(
"$JS.API.CONSUMER.CREATE.{events}.notify-router.{}",
@ -1820,6 +1827,12 @@ mod tests {
format!("$KV.{state}.42.5571"),
format!("$JS.API.DIRECT.GET.KV_{state}.$KV.{state}.42.5571"),
"$JS.API.STREAM.INFO.KV_agent-status".to_owned(),
// The flows document: read it.
format!("$JS.API.STREAM.INFO.KV_{flows}"),
format!(
"$JS.API.DIRECT.GET.KV_{flows}.$KV.{flows}.{}",
notify_flows::KEY
),
] {
assert!(may(&allowed), "the router may not reach {allowed}");
}
@ -1838,6 +1851,10 @@ mod tests {
forge_events::subject("issues"),
"$KV.agent-status.alpha.atlas".to_owned(),
"$KV.agent-schedules.atlas".to_owned(),
// The flows document: not write it, nor make the bucket.
format!("$KV.{flows}.{}", notify_flows::KEY),
format!("$JS.API.STREAM.CREATE.KV_{flows}"),
format!("$JS.API.STREAM.DELETE.KV_{flows}"),
swarm_queue_client::agent_input_subject("atlas"),
"$JS.API.STREAM.CREATE.KV_agent-status".to_owned(),
] {

View file

@ -163,6 +163,22 @@ mod tests {
}
}
/// The names a stored flow may read are the ones `fields` produces.
#[test]
fn the_fields_are_the_ones_a_stored_flow_may_name() {
let d = Delivery::parse(&issue_assign("assigned", &["atlas"])).expect("parses");
for event in ["issues", "issue_assign"] {
let mut got: Vec<&str> = fields(event, &d, &Snapshot::default())
.keys()
.copied()
.collect();
let mut want = swarm_wire::notify_flows::FIELDS.to_vec();
got.sort_unstable();
want.sort_unstable();
assert_eq!(got, want, "{event}");
}
}
#[test]
fn an_issue_delivery_reads_back_its_fields() {
let d = Delivery::parse(&issue_assign("assigned", &["atlas", "iris"])).expect("parses");

View file

@ -1,26 +1,17 @@
//! Flows: what the router does with an event.
//! Running a flow over one event.
//!
//! A flow is a graph of nodes joined by edges, rooted at one `source` node.
//! A `match` node passes when every one of its conditions holds, and then
//! each of its children runs, so one event can reach several branches. A
//! `sink` node is a leaf and emits one todo op per recipient.
//!
//! Flows are trees: one source, one parent for every other node, every node
//! reachable from the source. [`Flow::validate`] checks that, and the router
//! runs only flows that pass.
//!
//! The built-in flows ([`builtins`]) are ordinary flows defined in code.
//! The flow types, their checks and the built-in flows are
//! `swarm_wire::notify_flows`; the built-in and the custom flows are the same
//! type and this walk runs both. A `match` node passes when every one of its
//! conditions holds, and then each of its children runs, so one event can
//! reach several branches. A `todo` node emits one op per recipient. The
//! router runs only flows that pass [`Flow::validate`].
use std::collections::{BTreeMap, BTreeSet};
use std::collections::BTreeMap;
use swarm_wire::notify_flows::{Condition, Flow, Node, NodeKind, Recipients, TodoSink};
use swarm_wire::router_todo::{Op, RouterTodoOp};
/// The most nodes one flow may have.
const MAX_NODES: usize = 64;
/// The deepest a node may sit below the source.
const MAX_DEPTH: usize = 16;
/// One event, as the fields a flow reads.
pub type Fields = BTreeMap<&'static str, Value>;
@ -31,56 +22,6 @@ pub enum Value {
List(Vec<String>),
}
#[derive(Clone, Debug)]
pub struct Flow {
pub id: String,
pub nodes: Vec<Node>,
pub edges: Vec<Edge>,
}
#[derive(Clone, Debug)]
pub struct Node {
pub id: String,
pub kind: NodeKind,
}
/// A directed edge, parent to child.
#[derive(Clone, Debug)]
pub struct Edge {
pub from: String,
pub to: String,
}
#[derive(Clone, Debug)]
pub enum NodeKind {
/// The root: the forge event types the flow starts from.
Source { events: Vec<String> },
/// Passes when every condition holds.
Match { when: Vec<Condition> },
/// Emits a todo op to each recipient.
Todo(TodoSink),
}
/// `field` equals `value`. A list field holds when one of its items does.
#[derive(Clone, Debug)]
pub struct Condition {
pub field: String,
pub equals: String,
}
#[derive(Clone, Debug)]
pub struct TodoSink {
/// The field naming the recipients, a list of agent names.
pub to: String,
pub op: Op,
/// The todo key, with `{field}` placeholders.
pub key: String,
/// For a clear: `key` is a prefix.
pub prefix: bool,
/// The todo text, with `{field}` placeholders. Unused for a clear.
pub text: String,
}
/// One op the walk emitted, for one recipient.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct Action {
@ -90,130 +31,72 @@ pub struct Action {
pub op: RouterTodoOp,
}
impl Flow {
/// Check the flow is a tree within the caps, or say why not.
pub fn validate(&self) -> Result<(), String> {
if self.nodes.len() > MAX_NODES {
return Err(format!("{} nodes, more than {MAX_NODES}", self.nodes.len()));
}
let mut ids = BTreeSet::new();
for node in &self.nodes {
if !ids.insert(node.id.as_str()) {
return Err(format!("node id {} is used twice", node.id));
}
}
for edge in &self.edges {
for end in [&edge.from, &edge.to] {
if !ids.contains(end.as_str()) {
return Err(format!("an edge names {end}, which is no node"));
/// Walk `flow` for one event of type `event_type` and return the ops it
/// emits. Assumes [`Flow::validate`] passed.
pub fn evaluate(flow: &Flow, event_type: &str, fields: &Fields) -> Vec<Action> {
let Some(root) = flow
.nodes
.iter()
.find(|n| matches!(&n.kind, NodeKind::Source { events } if events.iter().any(|e| e == event_type)))
else {
return Vec::new();
};
let mut actions = Vec::new();
let mut stack = vec![root.id.as_str()];
while let Some(id) = stack.pop() {
let Some(node) = flow.node(id) else { continue };
match &node.kind {
NodeKind::Source { .. } => stack.extend(flow.children(id)),
NodeKind::Match { when } => {
if when.iter().all(|c| holds(c, fields)) {
stack.extend(flow.children(id));
}
}
NodeKind::Todo(sink) => actions.extend(emit(flow, node, sink, fields)),
}
let roots: Vec<&Node> = self
.nodes
.iter()
.filter(|n| matches!(n.kind, NodeKind::Source { .. }))
.collect();
let [root] = roots.as_slice() else {
return Err(format!("{} source nodes, not one", roots.len()));
};
for node in &self.nodes {
if matches!(node.kind, NodeKind::Todo(_)) && !self.children(&node.id).is_empty() {
return Err(format!("sink {} has children", node.id));
}
}
// From the one root, every node reached exactly once: a node reached
// twice has a second parent or sits on a cycle, and one not reached is
// cut off.
let mut reached = 0;
let mut stack = vec![(root.id.as_str(), 0)];
while let Some((id, depth)) = stack.pop() {
if depth > MAX_DEPTH {
return Err(format!("node {id} is deeper than {MAX_DEPTH}"));
}
reached += 1;
if reached > self.nodes.len() {
return Err(
"a node is reached twice: it has two parents or sits on a cycle".to_owned(),
);
}
stack.extend(self.children(id).into_iter().map(|c| (c, depth + 1)));
}
if reached != self.nodes.len() {
return Err(format!(
"{} of {} nodes are not reachable from the source",
self.nodes.len() - reached,
self.nodes.len()
));
}
Ok(())
}
actions
}
fn children(&self, id: &str) -> Vec<&str> {
self.edges
.iter()
.filter(|e| e.from == id)
.map(|e| e.to.as_str())
.collect()
}
fn node(&self, id: &str) -> Option<&Node> {
self.nodes.iter().find(|n| n.id == id)
}
/// Walk the flow for one event of type `event_type` and return the ops it
/// emits. Assumes [`Self::validate`] passed.
pub fn evaluate(&self, event_type: &str, fields: &Fields) -> Vec<Action> {
let Some(root) = self
.nodes
.iter()
.find(|n| matches!(&n.kind, NodeKind::Source { events } if events.iter().any(|e| e == event_type)))
else {
return Vec::new();
};
let mut actions = Vec::new();
let mut stack = vec![root.id.as_str()];
while let Some(id) = stack.pop() {
let Some(node) = self.node(id) else { continue };
match &node.kind {
NodeKind::Source { .. } => stack.extend(self.children(id)),
NodeKind::Match { when } => {
if when.iter().all(|c| holds(c, fields)) {
stack.extend(self.children(id));
}
}
NodeKind::Todo(sink) => actions.extend(self.emit(node, sink, fields)),
}
}
actions
}
fn emit(&self, node: &Node, sink: &TodoSink, fields: &Fields) -> Vec<Action> {
let recipients = match fields.get(sink.to.as_str()) {
fn emit(flow: &Flow, node: &Node, sink: &TodoSink, fields: &Fields) -> Vec<Action> {
let recipients = match &sink.to {
Recipients::Agent(agent) => vec![agent.clone()],
Recipients::Field(name) => match fields.get(name.as_str()) {
Some(Value::List(items)) => items.clone(),
Some(Value::Text(one)) => vec![one.clone()],
None => Vec::new(),
},
};
let rendered = expand(&sink.key, fields).and_then(|key| {
let summary = match sink.op {
Op::Upsert => expand(&sink.text, fields)?,
Op::Clear => String::new(),
};
let op = RouterTodoOp {
op: sink.op,
key: expand(&sink.key, fields),
prefix: sink.op == Op::Clear && sink.prefix,
summary: match sink.op {
Op::Upsert => expand(&sink.text, fields),
Op::Clear => String::new(),
},
source: format!("router:{}", self.id),
};
recipients
.into_iter()
.map(|agent| Action {
flow: self.id.clone(),
node: node.id.clone(),
agent,
op: op.clone(),
})
.collect()
}
Ok((key, summary))
});
let (key, summary) = match rendered {
Ok(done) => done,
Err(why) => {
tracing::warn!(flow = %flow.id, node = %node.id, %why, "notify router: a todo node names a placeholder the event lacks; nothing sent");
return Vec::new();
}
};
let op = RouterTodoOp {
op: sink.op,
key,
prefix: sink.op == Op::Clear && sink.prefix,
summary,
source: format!("router:{}", flow.id),
};
recipients
.into_iter()
.map(|agent| Action {
flow: flow.id.clone(),
node: node.id.clone(),
agent,
op: op.clone(),
})
.collect()
}
fn holds(condition: &Condition, fields: &Fields) -> bool {
@ -224,9 +107,9 @@ fn holds(condition: &Condition, fields: &Fields) -> bool {
}
}
/// `template` with each `{field}` replaced by that text field's value. A list
/// field is joined with `, `; an unknown name is left as written.
pub fn expand(template: &str, fields: &Fields) -> String {
/// `template` with each `{field}` replaced by that field's value, a list
/// joined with `, `. A name that is no field is an error naming it.
pub fn expand(template: &str, fields: &Fields) -> Result<String, String> {
let mut out = String::with_capacity(template.len());
let mut rest = template;
while let Some(open) = rest.find('{') {
@ -234,109 +117,24 @@ pub fn expand(template: &str, fields: &Fields) -> String {
let after = &rest[open + 1..];
let Some(close) = after.find('}') else {
out.push_str(&rest[open..]);
return out;
return Ok(out);
};
let name = &after[..close];
match fields.get(name) {
Some(Value::Text(v)) => out.push_str(v),
Some(Value::List(items)) => out.push_str(&items.join(", ")),
None => {
out.push('{');
out.push_str(name);
out.push('}');
}
None => return Err(format!("{{{name}}} is no event field")),
}
rest = &after[close + 1..];
}
out.push_str(rest);
out
}
/// The flows the router ships with.
///
/// - `assigned`: a newly assigned agent gets a todo for the issue or PR.
/// - `unassigned`: an agent taken off it gets that todo rewritten to say so.
/// Both write the same key, so the second replaces the first.
/// - `closed`: closing it clears every router todo the router queued for it.
pub fn builtins() -> Vec<Flow> {
let assignment_key = "forge:{repo}#{number}:assignment";
vec![
chain(
"assigned",
&["issue_assign", "pull_request_assign"],
"assigned",
TodoSink {
to: "assignees_added".to_owned(),
op: Op::Upsert,
key: assignment_key.to_owned(),
prefix: false,
text: "you are assigned to {kind} {repo}#{number}: {title} ({url})".to_owned(),
},
),
chain(
"unassigned",
&["issue_assign", "pull_request_assign"],
"unassigned",
TodoSink {
to: "assignees_removed".to_owned(),
op: Op::Upsert,
key: assignment_key.to_owned(),
prefix: false,
text: "you are not assigned to {kind} {repo}#{number} anymore: {title} ({url})"
.to_owned(),
},
),
chain(
"closed",
&["issues", "pull_request"],
"closed",
TodoSink {
to: "todo_holders".to_owned(),
op: Op::Clear,
key: "forge:{repo}#{number}:".to_owned(),
prefix: true,
text: String::new(),
},
),
]
}
/// source(`events`) → match(`action` = `action`) → `sink`.
fn chain(id: &str, events: &[&str], action: &str, sink: TodoSink) -> Flow {
let node = |id: &str, kind| Node {
id: id.to_owned(),
kind,
};
let edge = |from: &str, to: &str| Edge {
from: from.to_owned(),
to: to.to_owned(),
};
Flow {
id: id.to_owned(),
nodes: vec![
node(
"source",
NodeKind::Source {
events: events.iter().map(|e| (*e).to_owned()).collect(),
},
),
node(
"match",
NodeKind::Match {
when: vec![Condition {
field: "action".to_owned(),
equals: action.to_owned(),
}],
},
),
node("sink", NodeKind::Todo(sink)),
],
edges: vec![edge("source", "match"), edge("match", "sink")],
}
Ok(out)
}
#[cfg(test)]
mod tests {
use swarm_wire::notify_flows::{Edge, builtins};
use super::*;
fn text(v: &str) -> Value {
@ -368,16 +166,11 @@ mod tests {
.expect("a built-in")
}
#[test]
fn every_builtin_is_a_valid_tree() {
for f in builtins() {
assert_eq!(f.validate(), Ok(()), "{}", f.id);
}
}
#[test]
fn assigning_queues_a_todo_for_each_newly_assigned_agent_only() {
let got = flow("assigned").evaluate(
let assigned = flow("assigned");
let got = evaluate(
&assigned,
"issue_assign",
&event("assigned", &["atlas", "iris"], &[], &[]),
);
@ -392,21 +185,28 @@ mod tests {
assert_eq!(got[0].op.source, "router:assigned");
// The match gates: an unassign event reaches no assigned sink.
assert!(
flow("assigned")
.evaluate("issue_assign", &event("unassigned", &["atlas"], &[], &[]))
.is_empty()
evaluate(
&assigned,
"issue_assign",
&event("unassigned", &["atlas"], &[], &[])
)
.is_empty()
);
// The source gates: another event type starts nothing.
assert!(
flow("assigned")
.evaluate("issues", &event("assigned", &["atlas"], &[], &[]))
.is_empty()
evaluate(
&assigned,
"issues",
&event("assigned", &["atlas"], &[], &[])
)
.is_empty()
);
}
#[test]
fn unassigning_rewrites_the_same_key_for_the_removed_agent() {
let got = flow("unassigned").evaluate(
let got = evaluate(
&flow("unassigned"),
"pull_request_assign",
&event("unassigned", &[], &["argus"], &[]),
);
@ -423,8 +223,12 @@ mod tests {
#[test]
fn closing_clears_by_prefix_for_every_holder() {
let got =
flow("closed").evaluate("issues", &event("closed", &[], &[], &["atlas", "argus"]));
let closed = flow("closed");
let got = evaluate(
&closed,
"issues",
&event("closed", &[], &[], &["atlas", "argus"]),
);
assert_eq!(got.len(), 2);
for a in &got {
assert_eq!(a.op.op, Op::Clear);
@ -432,11 +236,7 @@ mod tests {
assert_eq!(a.op.key, "forge:hyperhive/hyperhive#7:");
assert_eq!(a.op.summary, "");
}
assert!(
flow("closed")
.evaluate("issues", &event("reopened", &[], &[], &["atlas"]))
.is_empty()
);
assert!(evaluate(&closed, "issues", &event("reopened", &[], &[], &["atlas"])).is_empty());
}
/// A match node with two children runs both: the branching the tree
@ -450,7 +250,7 @@ mod tests {
f.nodes.push(Node {
id: "second".to_owned(),
kind: NodeKind::Todo(TodoSink {
to: "assignees_removed".to_owned(),
to: Recipients::Field("assignees_removed".to_owned()),
..sink
}),
});
@ -459,7 +259,8 @@ mod tests {
to: "second".to_owned(),
});
assert_eq!(f.validate(), Ok(()));
let got = f.evaluate(
let got = evaluate(
&f,
"issue_assign",
&event("assigned", &["atlas"], &["argus"], &[]),
);
@ -471,48 +272,59 @@ mod tests {
assert_eq!(nodes, [("second", "argus"), ("sink", "atlas")]);
}
/// A custom flow's sink can name one agent, whatever the event holds.
#[test]
fn a_flow_that_is_not_a_tree_is_refused() {
let edge = |from: &str, to: &str| Edge {
from: from.to_owned(),
to: to.to_owned(),
};
let mut cycle = flow("assigned");
cycle.edges[0] = edge("sink", "match");
cycle.edges.push(edge("match", "sink"));
assert!(cycle.validate().is_err(), "a cycle");
let mut orphan = flow("assigned");
orphan.edges.pop();
assert!(orphan.validate().is_err(), "an unreachable sink");
let mut two_parents = flow("assigned");
two_parents.edges.push(edge("source", "sink"));
assert!(two_parents.validate().is_err(), "a sink with two parents");
let mut sink_child = flow("assigned");
sink_child.nodes.push(Node {
id: "m2".to_owned(),
kind: NodeKind::Match { when: Vec::new() },
fn a_sink_to_a_named_agent_reaches_it_once() {
let mut f = flow("assigned");
f.nodes[2].kind = NodeKind::Todo(TodoSink {
to: Recipients::Agent("triage".to_owned()),
op: Op::Upsert,
key: "forge:{repo}#{number}:triage".to_owned(),
prefix: false,
text: "triage {repo}#{number}".to_owned(),
});
sink_child.edges.push(edge("sink", "m2"));
assert!(sink_child.validate().is_err(), "a sink with a child");
let mut dangling = flow("assigned");
dangling.edges.push(edge("match", "nowhere"));
assert!(dangling.validate().is_err(), "an edge to no node");
let mut two_sources = flow("assigned");
two_sources.nodes[1].kind = NodeKind::Source { events: Vec::new() };
assert!(two_sources.validate().is_err(), "two sources");
let got = evaluate(
&f,
"issue_assign",
&event("assigned", &["atlas", "iris"], &[], &[]),
);
assert_eq!(got.len(), 1);
assert_eq!(got[0].agent, "triage");
assert_eq!(got[0].op.summary, "triage hyperhive/hyperhive#7");
assert_eq!(got[0].op.source, "router:assigned");
}
#[test]
fn a_placeholder_expands_to_its_field_and_an_unknown_one_stays() {
fn a_placeholder_expands_to_its_field_and_an_unknown_one_is_an_error() {
let fields = event("assigned", &["a", "b"], &[], &[]);
assert_eq!(
expand("{repo}#{number} {assignees_added} {nope} {", &fields),
"hyperhive/hyperhive#7 a, b {nope} {"
expand("{repo}#{number} {assignees_added} {", &fields),
Ok("hyperhive/hyperhive#7 a, b {".to_owned())
);
assert_eq!(
expand("x {nope}", &fields),
Err("{nope} is no event field".to_owned())
);
}
/// A sink naming a placeholder the event lacks sends nothing, rather
/// than a todo under a literal `{name}` key.
#[test]
fn a_sink_with_an_unknown_placeholder_emits_nothing() {
let mut f = flow("assigned");
f.nodes[2].kind = NodeKind::Todo(TodoSink {
to: Recipients::Agent("triage".to_owned()),
op: Op::Upsert,
key: "forge:{repo}#{numbr}:triage".to_owned(),
prefix: false,
text: "t".to_owned(),
});
let fields = event("assigned", &["atlas"], &[], &[]);
assert!(evaluate(&f, "issue_assign", &fields).is_empty());
// Control: spelled right, the same flow emits.
if let NodeKind::Todo(s) = &mut f.nodes[2].kind {
s.key = "forge:{repo}#{number}:triage".to_owned();
}
assert_eq!(evaluate(&f, "issue_assign", &fields).len(), 1);
}
}

View file

@ -4,7 +4,8 @@
//! (`swarm_wire::forge_events`), runs every flow over each delivery
//! ([`flow`], [`route`]), and queues the todo ops the flows emit on each
//! recipient agent's subject of `ROUTER_TODOS` (`swarm_wire::router_todo`).
//! Each agent's harness applies them to its own todo list.
//! Each agent's harness applies them to its own todo list. The flows are the
//! built-in ones, changed by the `notify-flows` document ([`stored`]).
//!
//! Per issue it keeps a snapshot in its state bucket ([`event::Snapshot`]):
//! the assignees it last saw and the agents it queued a todo for. An event is
@ -31,6 +32,7 @@ mod event;
mod flow;
mod route;
mod settle;
mod stored;
/// The cert-auth role this service logs in to the store as. `swarm-bao.nix`
/// writes it, matching the leaf's subject.
@ -83,17 +85,7 @@ async fn main() -> Result<()> {
})
.await?;
let flows: Vec<flow::Flow> = flow::builtins()
.into_iter()
.filter(|f| match f.validate() {
Ok(()) => true,
Err(reason) => {
tracing::error!(flow = %f.id, %reason, "notify router: skipping a flow that is not a tree");
false
}
})
.collect();
let mut router = Router::open(&queue, flows).await?;
let mut router = Router::open(&queue).await?;
tracing::info!(
stream = forge_events::STREAM,
"notify router: routing forge events"
@ -113,12 +105,12 @@ struct Router {
js: jetstream::Context,
events: jetstream::consumer::Consumer<pull::Config>,
state: kv::Store,
flows: Vec<flow::Flow>,
flows: stored::Flows,
roster: Option<(Instant, BTreeSet<String>)>,
}
impl Router {
async fn open(queue: &async_nats::Client, flows: Vec<flow::Flow>) -> Result<Self> {
async fn open(queue: &async_nats::Client) -> Result<Self> {
let js = jetstream::new(queue.clone());
let events = swarm_queue_client::forge_events::open_or_create(queue)
.await
@ -154,7 +146,7 @@ impl Router {
js,
events,
state,
flows,
flows: stored::Flows::builtins(),
roster: None,
})
}
@ -193,8 +185,9 @@ impl Router {
}),
None => event::Snapshot::default(),
};
self.flows.refresh(&self.js).await;
let roster = self.roster().await.map_err(Skip::Retry)?;
let routed = route::route(&event_type, &delivery, &snapshot, &self.flows, |a| {
let routed = route::route(&event_type, &delivery, &snapshot, &self.flows.active, |a| {
roster.contains(a)
});
for action in &routed.actions {

View file

@ -6,7 +6,9 @@ use std::collections::BTreeSet;
use swarm_wire::router_todo::Op;
use crate::event::{Delivery, Snapshot, fields, is_assign};
use crate::flow::{Action, Flow};
use swarm_wire::notify_flows::Flow;
use crate::flow::{Action, evaluate};
/// What one event comes to.
#[derive(Debug, PartialEq, Eq)]
@ -30,7 +32,7 @@ pub fn route(
let fields = fields(event_type, delivery, snapshot);
let mut actions = Vec::new();
let mut skipped = Vec::new();
for action in flows.iter().flat_map(|f| f.evaluate(event_type, &fields)) {
for action in flows.iter().flat_map(|f| evaluate(f, event_type, &fields)) {
if is_agent(&action.agent) {
actions.push(action);
} else if !skipped.contains(&action.agent) {
@ -68,7 +70,7 @@ pub fn route(
#[cfg(test)]
mod tests {
use super::*;
use crate::flow::builtins;
use swarm_wire::notify_flows::builtins;
fn delivery(action: &str, assignees: &[&str]) -> Delivery {
Delivery {

View file

@ -0,0 +1,259 @@
//! The flows the router runs: the built-ins, changed by the document in the
//! `notify-flows` bucket.
//!
//! The router reads the key again every [`REFRESH`] and runs what
//! [`Document::active`] says. What it does with each thing it can read:
//!
//! | the key | the router runs |
//! | ------------------------------------ | ----------------------------------- |
//! | absent, deleted, or no such bucket | the built-ins |
//! | a document it reads | its built-ins and custom flows |
//! | a document it cannot read | what it ran before |
//! | the read itself failing | what it ran before |
//!
//! "Cannot read" is not JSON, not the document's shape, or a version this
//! build does not know. Before any document has been read, "what it ran
//! before" is the built-ins, so a bad document never leaves the router with
//! no flows.
use std::time::{Duration, Instant};
use async_nats::jetstream::ErrorCode;
use async_nats::jetstream::context::GetStreamErrorKind;
use async_nats::jetstream::{self, kv};
use swarm_wire::notify_flows::{self, Document, Flow};
/// How long a read of the key is reused.
const REFRESH: Duration = Duration::from_secs(10);
/// What a read of the key came to.
#[derive(Debug)]
pub enum Read {
/// No key, a deleted key, or no bucket.
Absent,
Document {
revision: u64,
bytes: Vec<u8>,
},
Failed(String),
}
/// The flows in force, and which revision of the key they came from.
pub struct Flows {
pub active: Vec<Flow>,
/// The revision `active` was read from; `None` for the built-ins alone.
revision: Option<u64>,
/// The last problem logged, so a standing one is logged once.
problem: Option<String>,
checked: Option<Instant>,
}
impl Flows {
/// The built-ins, which is what the router runs until it reads a document.
pub fn builtins() -> Self {
Self {
active: Document::default().active().0,
revision: None,
problem: None,
checked: None,
}
}
/// Take what a read came to.
pub fn apply(&mut self, read: Read) {
match read {
Read::Failed(why) => self.note(format!("reading the key failed: {why}")),
Read::Absent => {
if self.revision.take().is_some() {
tracing::info!(
"notify router: the flows document is gone; running the built-in flows"
);
}
self.active = Document::default().active().0;
self.problem = None;
}
Read::Document { revision, .. } if self.revision == Some(revision) => {}
Read::Document { revision, bytes } => match Document::decode(&bytes) {
Err(why) => {
// The revision is not taken, so the next read looks at
// the document again and the problem is logged once.
self.note(format!("revision {revision} cannot be read: {why}"));
}
Ok(doc) => {
let (active, skipped) = doc.active();
for why in &skipped {
tracing::error!(%why, "notify router: skipping a flow");
}
tracing::info!(
revision,
flows = active.len(),
disabled = doc.disabled_builtins.len(),
custom = doc.flows.len(),
"notify router: running the flows document"
);
self.active = active;
self.revision = Some(revision);
self.problem = None;
}
},
}
}
fn note(&mut self, problem: String) {
if self.problem.as_ref() != Some(&problem) {
tracing::error!(%problem, "notify router: keeping the flows it has");
self.problem = Some(problem);
}
}
/// Read the key if the last read is older than [`REFRESH`].
pub async fn refresh(&mut self, js: &jetstream::Context) {
if self.checked.is_some_and(|at| at.elapsed() < REFRESH) {
return;
}
self.checked = Some(Instant::now());
self.apply(read(js).await);
}
}
async fn read(js: &jetstream::Context) -> Read {
// A bucket that does not exist is the answer "no document". Any other
// failure to look, such as a timeout, says nothing about the document.
match js.get_stream(format!("KV_{}", notify_flows::BUCKET)).await {
Ok(_) => {}
Err(e)
if matches!(e.kind(), GetStreamErrorKind::JetStream(j)
if j.error_code() == ErrorCode::STREAM_NOT_FOUND) =>
{
return Read::Absent;
}
Err(e) => return Read::Failed(e.to_string()),
}
let store = match js.get_key_value(notify_flows::BUCKET).await {
Ok(store) => store,
Err(e) => return Read::Failed(e.to_string()),
};
match store.entry(notify_flows::KEY).await {
Ok(Some(entry)) if entry.operation == kv::Operation::Put => Read::Document {
revision: entry.revision,
bytes: entry.value.to_vec(),
},
Ok(_) => Read::Absent,
Err(e) => Read::Failed(e.to_string()),
}
}
#[cfg(test)]
mod tests {
use swarm_wire::notify_flows::{Document, VERSION};
use super::*;
fn ids(f: &Flows) -> Vec<&str> {
f.active.iter().map(|f| f.id.as_str()).collect()
}
fn doc(disabled: &[&str], custom: &[&str]) -> Read {
let mut flows = Vec::new();
for id in custom {
let mut f = notify_flows::builtins().remove(0);
f.id = (*id).to_owned();
flows.push(f);
}
let d = Document {
version: VERSION,
disabled_builtins: disabled.iter().map(|s| (*s).to_owned()).collect(),
flows,
};
Read::Document {
revision: 0,
bytes: serde_json::to_vec(&d).unwrap(),
}
}
fn at(revision: u64, read: Read) -> Read {
match read {
Read::Document { bytes, .. } => Read::Document { revision, bytes },
other => other,
}
}
fn junk(revision: u64, bytes: &[u8]) -> Read {
Read::Document {
revision,
bytes: bytes.to_vec(),
}
}
#[test]
fn with_no_document_the_router_runs_the_builtins() {
let mut f = Flows::builtins();
assert_eq!(ids(&f), ["assigned", "unassigned", "closed"]);
f.apply(Read::Absent);
assert_eq!(ids(&f), ["assigned", "unassigned", "closed"]);
// An empty document is the same.
f.apply(at(1, doc(&[], &[])));
assert_eq!(ids(&f), ["assigned", "unassigned", "closed"]);
}
#[test]
fn a_document_switches_builtins_off_and_adds_custom_flows() {
let mut f = Flows::builtins();
f.apply(at(1, doc(&["unassigned"], &["triage"])));
assert_eq!(ids(&f), ["assigned", "closed", "triage"]);
}
#[test]
fn a_document_it_cannot_read_leaves_the_flows_as_they_were() {
let mut f = Flows::builtins();
f.apply(at(1, doc(&["closed"], &["triage"])));
let before = ids(&f).join(",");
assert_eq!(before, "assigned,unassigned,triage");
for bad in [
junk(2, b"not json"),
junk(3, br#"{"version":2,"flows":[]}"#),
junk(4, br#"{"flows":[{"id":"x"}]}"#),
Read::Failed("timeout".to_owned()),
] {
f.apply(bad);
assert_eq!(ids(&f).join(","), before);
}
// The next good revision replaces them: the control that the bad
// ones were kept out by being bad, not by the loader being stuck.
f.apply(at(5, doc(&[], &[])));
assert_eq!(ids(&f), ["assigned", "unassigned", "closed"]);
}
#[test]
fn a_bad_first_document_is_the_builtins_not_nothing() {
let mut f = Flows::builtins();
f.apply(junk(1, br#"{"version":9}"#));
assert_eq!(ids(&f), ["assigned", "unassigned", "closed"]);
}
#[test]
fn a_deleted_document_goes_back_to_the_builtins() {
let mut f = Flows::builtins();
f.apply(at(1, doc(&["closed"], &["triage"])));
f.apply(Read::Absent);
assert_eq!(ids(&f), ["assigned", "unassigned", "closed"]);
}
#[test]
fn a_flow_that_is_not_valid_is_left_out_and_the_rest_run() {
let mut f = Flows::builtins();
let mut broken = notify_flows::builtins().remove(0);
broken.id = "broken".to_owned();
broken.edges.clear();
let d = Document {
version: VERSION,
disabled_builtins: Vec::new(),
flows: vec![broken],
};
f.apply(Read::Document {
revision: 1,
bytes: serde_json::to_vec(&d).unwrap(),
});
assert_eq!(ids(&f), ["assigned", "unassigned", "closed"]);
}
}

View file

@ -18,6 +18,7 @@ pub mod builder;
pub mod claude_login;
pub mod forge_events;
pub mod notices;
pub mod notify_flows;
pub mod notify_router;
pub mod review_classifier;
pub mod router_todo;

Some files were not shown because too many files have changed in this diff Show more