# Notification router The swarm's `swarm-notify-router` service turns forge events into todos on the agents they concern. No hive is in the path, and the agent can be on any hive: swarm-controller keeps the events, the router decides, and each agent's harness takes its own todos off the queue. Where it runs and how it's enabled: [`services.md`](services.md#notification-router). It runs beside `hive-forge-notify`, which still delivers forge notifications the way it did. An agent can get the same news from both. ## What an agent gets | Flow | Event | The agent's todo | | ------------ | ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `assigned` | someone assigns an agent to an issue or PR | `you are assigned to issue #: (<url>)`, key `forge:<repo>#<n>:assignment` | | `unassigned` | someone takes an agent off an issue or PR | the same key rewritten: `you are not assigned to issue <repo>#<n> anymore: <title> (<url>)` | | `closed` | someone closes or merges the issue or PR | the router clears every router todo it queued for it, then queues `closed` on each agent that held one: `<kind> <repo>#<n> was closed: <title> (<url>)`, key `forge:<repo>#<n>:closed` | The todos are subsystem `router` in the agent's todo list. A todo under the same key replaces the one before, so an issue gives an agent one assignment todo however often it changes hands. A new event reopens a todo the agent marked done. The `closed` todo has its own key, apart from the assignment todo. A reopened issue leaves it in place until the next close replaces it; a new assignment writes the `assignment` key and leaves it alone. Only agents get todos. An assignee with no agent of that name gets nothing. An agent assigned before it first reports its status gets its todo on the issue's next assign event. Forgejo's assign events carry the assignees after the change, not which one changed. The router keeps the assignees it saw on the last assign event per issue and takes the difference. With no earlier assign event for an issue, every current assignee counts as newly assigned and nobody as removed: an router notices an unassign only on an issue it has seen an assign event for. ## The path 1. swarm-controller registers two instance-wide hooks, one for the `issues` and `issue_assign` events at `/webhook/forge/issues` and one for `pull_request`, `pull_request_assign` and `pull_request_sync` (a push to an open PR's branch) at `/webhook/forge/pull-request`. At each start it adds any of these events a registered hook lacks. 2. On each delivery the controller checks the HMAC and publishes the body, unchanged, to the `SWARM_EVENTS` stream on `$SWARM.events.forge.<X-Forgejo-Event-Type>`, such as `$SWARM.events.forge.issue_assign`. The stream keeps a delivery 7 days and drops a redelivery by its `X-Forgejo-Delivery` id. A delivery the stream did not keep answers 503, so the forge's delivery log shows it. 3. The router drains its durable consumer, `notify-router`, in order. For each event it runs every flow, queues the todo ops the flows emit, writes the issue's snapshot to its `notify-router-state` bucket, then acks the event. A failed step stops the batch: that event and the rest come back in stream order after 30 seconds, and the stream drops the ops queued twice by their message id. 4. Each op goes to the `ROUTER_TODOS` stream on the agent's own subject, `$SWARM.router.todo.<agent>`. Work-queue retention: an op stays until its agent acks it, up to 30 days and 1000 per agent. 5. The agent's harness creates its own durable consumer, `todo-<agent>`, applies each op to its todo list and acks it. An agent that's down, or whose harness predates this, leaves its ops waiting. The router holds no forge token and no webhook secret, and registers no hook. ## Flows A flow is a graph of nodes joined by edges: one `source` node naming the forge event types it starts from, `match` nodes that pass when every condition holds, `todo` nodes that emit one todo op per recipient, and `request_reviewer` nodes that ask for a review. A match node can have more than one child, and each one runs, so one event can reach several leaves. A flow must be a tree: one source, one parent for every other node, every node reachable, no children under a `todo` or a `request_reviewer`. The most is 64 nodes, 16 levels deep, and 100 custom flows. A `todo` node sends to the agents an event field holds (`assignees_added`, `todo_holders`, …) or to one named agent. The router ships the three flows above, each a source → match on `action` → todo, and `head-classifier`: source `pull_request_sync` → `request_reviewer` for `review-classifier` ([below](#requesting-a-review)). The code defines them (`swarm_wire::notify_flows::builtins`), so a build that changes one reaches every swarm. Each switches off on `/notifications`. A custom flow has the same type and the same evaluator, so it can do anything a built-in does. **The stored document.** swarm-controller keeps the rest in one document, the `all` key of the `notify-flows` bucket: the ids of the built-in flows you switched off, and the custom flows. Only the controller writes it, through `GET` and `PUT /api/notify/flows`. A `PUT` replaces the whole document in one compare-and-set. The controller answers 400 unless every flow passes the tree check above and reads only what exists: a source event from `issues`, `issue_assign`, `pull_request`, `pull_request_assign` or `pull_request_sync`, and condition and recipient fields from the event's field list. The same goes for the `{field}` placeholders in a todo's key and text. A `PUT` must carry the `revision` it read, `0` for no document, and gets 409 when the stored revision is another. Swarm UI's [`/notifications`](ui.md#notification-flows) page shows the document and switches built-ins. **The schema.** `GET /api/notify/flows` also returns a `schema` object, built from the constants the check reads, so an editor offers only what a `PUT` accepts: - `source_events`: `[{name, description}]`, the source events. - `fields`: `[{name, description}]`, the condition and recipient fields, and the `{field}` placeholders. - `node_kinds`: `[{kind, description, fields}]` for `source`, `match`, `todo` and `request_reviewer`; each field is `{name, description, required, values}`, and `values` lists the accepted `{name, description}` pairs (the `upsert` and `clear` of a todo's `op`), empty for other fields. - `limits`: `{max_nodes, max_depth, max_flows}`. A controller that predates `schema` leaves it out of the answer; a client reads it as optional. **Requesting a review.** A `request_reviewer` node has one field, `reviewer`, a forge login. For the PR the event concerns (`repo` and `number`; an event about an issue sends nothing), the router sends a NATS request on `$SWARM.router.sink.request-reviewer` with `{repo, number, reviewer}` and swarm-controller answers with one line: | Reply | Meaning | | --------------- | ---------------------------------------------------------------------------------- | | `requested` | the forge accepted the request | | `skipped:<why>` | the reviewer's review of the PR's head already stands; the controller sent nothing | | `error:<msg>` | the forge refused or didn't answer, or the request didn't decode | The controller asks the forge under its own token, which the router never holds. It skips a reviewer whose latest review is neither stale (the PR's head moved past the commit it reviewed) nor dismissed, because requesting a review from someone who already submitted one makes the forge dismiss it, approval included. The controller asks again once the review is older than the head. `hive-forge pr assign-reviewer` applies the same rule (`hive_types::review`). The router sends the request once, after it queued the event's todo ops and wrote the snapshot, and logs the answer. A controller that's down or answers `error:` doesn't hold up the event or any agent's todos, and the request is not retried: the next push asks again. A request repeated for the same head is harmless: a reviewer with a pending request is a no-op for the forge, and one who has reviewed gets `skipped`. `head-classifier` requests the review classifier's review on a push to any open PR. The classifier then runs as it does for any requested review ([`review-classifier.md`](review-classifier.md#trigger-and-lifecycle)), through its own request queue, which replaces a waiting request for the same PR. A repo the classifier's account can't read, or a PR that account opened, answers `error:`. **What the router runs.** It reads the key at most every 10 seconds, when an event arrives, and runs the built-ins you left on, then the custom flows that are on and pass the tree check. It logs a custom flow that fails the check, leaves it out, and runs the rest. | the key | the router runs | | -------------------------------------- | ----------------------- | | absent, deleted, or its bucket missing | the built-in flows | | a document it reads | the above | | a document it can't read | the flows it ran before | | a read that fails | the flows it ran before | "Can't read" means not JSON, not a flows document, or a `version` other than 1. Before the router has read any document, "before" means the built-ins, so a bad document never leaves it with no flows. The controller refuses to overwrite a document of a version it doesn't know. **Deploying in any order.** The controller, the router and the UI can be on different hosts. A router that predates `request_reviewer` reads a document holding such a node (a custom flow with one) as one it can't read and keeps the flows it had; it also has no `head-classifier`. A router that has it, with a controller that predates it, gets no answer to the request and logs that. A controller that predates this leaves the key absent, and the router runs the built-ins as before. A router that predates this ignores the document, so the page and the API show a state the router isn't yet running. ## Deliveries and dry runs **The deliveries log.** The router appends one row to the `NOTIFY_ROUTER_LOG` stream for each delivery it makes, and the stream keeps the newest 500. A delivery is a todo op queued for an agent or a review requested from the controller, whether it went through or not. A row has: | Field | Holds | | ------------------------- | ---------------------------------------------------------------------------------- | | `at` | Unix seconds | | `flow`, `node` | the flow and the node that delivered | | `event`, `repo`, `number` | the forge event type and the issue or PR it concerns | | `kind`, `target` | `todo` and the agent, or `request_reviewer` and the reviewer | | `ok`, `detail` | whether it went through, and the op and key, the controller's answer, or the error | A review the controller skipped (`skipped: <why>`) is `ok`: nothing was wrong. An event the router retries after a failed step writes its rows again, so a retry shows twice, and the page says so under the table. A row that takes longer than 2 seconds to write counts as failed: the router logs a warning and the delivery goes on. **The dry run.** The router runs a flows document over the newest 50 events of the `SWARM_EVENTS` stream and returns, for each flow the document turns on, how many of those events reached each node. A `match` node counts the events that passed it. It uses the walk the router delivers with and delivers nothing: no todo op, no review request, no log row and no snapshot written. It keeps its own per-issue snapshots, starting empty, so the first assign event it replays for an issue counts every assignee as newly added. Every recipient counts as an agent. **The API.** The router answers two NATS requests from swarm-controller, which serves them next to `GET /api/notify/flows`: | Route | Router subject | Answers | | ---------------------------- | ------------------------------ | ------------------------------------------------------------------ | | `GET /api/notify/deliveries` | `$SWARM.router.api.deliveries` | the newest rows, newest first; `?limit=` up to 500, 100 by default | | `GET /api/notify/dry-run` | `$SWARM.router.api.dry-run` | the saved flows document, run over the newest 50 events | The dry-run request carries a whole document, so a draft runs the same way. The controller sends the saved one. [`/notifications`](ui.md#notification-flows) shows both: the recent deliveries, and the counts beside each node. **Deploying in any order.** A router that predates these subjects doesn't subscribe to them, and the queue answers the controller's request at once with no responders. Both routes then answer 503 saying the router doesn't answer and may be an older build, and the page shows that in place of the table and the counts. A request the router hears and doesn't answer within 10 seconds is the same 503. A router that answers with a refusal, or with something the controller can't read, is 502. A controller that predates the routes has no route: the page, if newer, shows the failure. ## Credentials and grants | Principal | Grant | | --------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `swarm-notify-router`, in its container | store: `read` on `swarm/services/<router client id>/oidc/client`. Queue: its own consumer on `SWARM_EVENTS` (create, pull, ack) and `STREAM.MSG.GET` on it for the dry run, publish on `$SWARM.router.todo.*` and on `$SWARM.router.sink.request-reviewer`, create `ROUTER_TODOS`, create `NOTIFY_ROUTER_LOG`, publish its one subject and read it by sequence, its own `notify-router-state` bucket, read on the `notify-flows` bucket, `STREAM.INFO` on the agent-status bucket, which lists the agents, and subscribe to the two `$SWARM.router.api.*` request subjects and answer each once, for 10 seconds | | swarm-controller | publish on `$SWARM.events.forge.*` and on the two `$SWARM.router.api.*` request subjects; subscribe to `$SWARM.router.sink.request-reviewer` and answer each request once, for 30 seconds; creates and writes the `notify-flows` bucket (every bucket, by its `$KV.*.>` grant) | | an agent, under its own credential | create its own consumer `todo-<agent>` in the form whose filter is `$SWARM.router.todo.<agent>`, pull from it, and ack what it delivered | No other client may publish on `$SWARM.router.todo.*`, `$SWARM.router.log`, the two `$SWARM.router.api.*` subjects, `$SWARM.router.sink.request-reviewer` or `$SWARM.events.forge.*`, so each has one writer, and the controller alone hears the review requests and the router alone hears the api requests. The review request needs no secret of its own: the controller asks the forge under the token it already holds. The agent's create subject carries the consumer's filter, and the queue server checks it against the request, so an agent can't create a consumer that delivers another agent's todos.