243 lines
18 KiB
Markdown
243 lines
18 KiB
Markdown
# Notification router
|
|
|
|
The swarm's `swarm-notify-router` service turns forge events into todos on the
|
|
agents they concern. No hive is in the path, and the agent can be on any hive:
|
|
swarm-controller keeps the events, the router decides, and each agent's harness
|
|
takes its own todos off the queue.
|
|
|
|
Where it runs and how it's enabled: [`services.md`](services.md#notification-router).
|
|
|
|
It runs beside `hive-forge-notify`, which still delivers forge notifications
|
|
the way it did. An agent can get the same news from both.
|
|
|
|
## What an agent gets
|
|
|
|
| Flow | Event | The agent's todo |
|
|
| ------------ | ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| `assigned` | someone assigns an agent to an issue or PR | `you are assigned to issue <repo>#<n>: <title> (<url>)`, key `forge:<repo>#<n>:assignment` |
|
|
| `unassigned` | someone takes an agent off an issue or PR | the same key rewritten: `you are not assigned to issue <repo>#<n> anymore: <title> (<url>)` |
|
|
| `closed` | someone closes or merges the issue or PR | the router clears every router todo it queued for it, then queues `closed` on each agent that held one: `<kind> <repo>#<n> was closed: <title> (<url>)`, key `forge:<repo>#<n>:closed` |
|
|
|
|
The todos are subsystem `router` in the agent's todo list. A todo under the
|
|
same key replaces the one before, so an issue gives an agent one assignment
|
|
todo however often it changes hands. A new event reopens a todo the agent
|
|
marked done.
|
|
|
|
The `closed` todo has its own key, apart from the assignment todo. A reopened
|
|
issue leaves it in place until the next close replaces it; a new assignment
|
|
writes the `assignment` key and leaves it alone.
|
|
|
|
Only agents get todos. An assignee with no agent of that name gets nothing.
|
|
An agent assigned before it first reports its status gets its todo on the
|
|
issue's next assign event.
|
|
|
|
Forgejo's assign events carry the assignees after the change, not which one
|
|
changed. The router keeps the assignees it saw on the last assign event per
|
|
issue and takes the difference. With no earlier assign event for an issue,
|
|
every current assignee counts as newly assigned and nobody as removed: an
|
|
router notices an unassign only on an issue it has seen an assign event for.
|
|
|
|
## The path
|
|
|
|
1. swarm-controller registers two instance-wide hooks, one for the `issues`
|
|
and `issue_assign` events at `/webhook/forge/issues` and one for
|
|
`pull_request`, `pull_request_assign` and `pull_request_sync` (a push to an
|
|
open PR's branch) at `/webhook/forge/pull-request`. At each start it adds
|
|
any of these events a registered hook lacks.
|
|
2. On each delivery the controller checks the HMAC and publishes the body,
|
|
unchanged, to the `SWARM_EVENTS` stream on
|
|
`$SWARM.events.forge.<X-Forgejo-Event-Type>`, such as
|
|
`$SWARM.events.forge.issue_assign`. The stream keeps a delivery 7 days and
|
|
drops a redelivery by its `X-Forgejo-Delivery` id. A delivery the stream did
|
|
not keep answers 503, so the forge's delivery log shows it.
|
|
3. The router drains its durable consumer, `notify-router`, in order. For each
|
|
event it runs every flow, queues the todo ops the flows emit, writes the
|
|
issue's snapshot to its `notify-router-state` bucket, then acks the event. A
|
|
failed step stops the batch: that event and the rest come back in stream
|
|
order after 30 seconds, and the stream drops the ops queued twice by their
|
|
message id.
|
|
4. Each op goes to the `ROUTER_TODOS` stream on the agent's own subject,
|
|
`$SWARM.router.todo.<agent>`. Work-queue retention: an op stays until its
|
|
agent acks it, up to 30 days and 1000 per agent.
|
|
5. The agent's harness creates its own durable consumer, `todo-<agent>`,
|
|
applies each op to its todo list and acks it. An agent that's down, or
|
|
whose harness predates this, leaves its ops waiting.
|
|
|
|
The router holds no forge token and no webhook secret, and registers no hook.
|
|
|
|
## Flows
|
|
|
|
A flow is a graph of nodes joined by edges: one `source` node naming the forge
|
|
event types it starts from, `match` nodes that pass when every condition
|
|
holds, `todo` nodes that emit one todo op per recipient, and `request_reviewer`
|
|
nodes that ask for a review. A match node can have more than one child, and
|
|
each one runs, so one event can reach several leaves. A flow must be a tree:
|
|
one source, one parent for every other node, every node reachable, no children
|
|
under a `todo` or a `request_reviewer`. The most is 64 nodes, 16
|
|
levels deep, and 100 custom flows. A `todo` node sends to the agents an event
|
|
field holds (`assignees_added`, `todo_holders`, …) or to one named agent.
|
|
|
|
The router ships the three flows above, each a source → match on `action` →
|
|
todo, and `head-classifier`: source `pull_request_sync` → `request_reviewer`
|
|
for `review-classifier` ([below](#requesting-a-review)). The code defines them
|
|
(`swarm_wire::notify_flows::builtins`), so a build that changes one reaches
|
|
every swarm. Each switches off on `/notifications`. A custom flow has the same type and
|
|
the same evaluator, so it can do anything a built-in does.
|
|
|
|
**The stored document.** swarm-controller keeps the rest in one document, the
|
|
`all` key of the `notify-flows` bucket: the ids of the built-in flows you
|
|
switched off, and the custom flows. Only the controller writes it, through
|
|
`GET` and `PUT /api/notify/flows`. A `PUT` replaces the whole document in one
|
|
compare-and-set. The controller answers 400 unless every flow passes the tree
|
|
check above and reads only what exists: a source event from `issues`,
|
|
`issue_assign`, `pull_request`, `pull_request_assign` or `pull_request_sync`,
|
|
and condition and recipient fields from the event's field list. The same goes for the `{field}`
|
|
placeholders in a todo's key and text. A `PUT` must carry the `revision` it read, `0` for no
|
|
document, and gets 409 when the stored revision is another. Swarm UI's
|
|
[`/notifications`](ui.md#notification-flows) page shows the document and
|
|
switches built-ins.
|
|
|
|
**The schema.** `GET /api/notify/flows` also returns a `schema` object, built
|
|
from the constants the check reads, so an editor offers only what a `PUT`
|
|
accepts:
|
|
|
|
- `source_events`: `[{name, description}]`, the source events.
|
|
- `fields`: `[{name, description}]`, the condition and recipient fields, and
|
|
the `{field}` placeholders.
|
|
- `node_kinds`: `[{kind, description, fields}]` for `source`, `match`, `todo`
|
|
and `request_reviewer`; each field is `{name, description, required, values}`, and `values`
|
|
lists the accepted `{name, description}` pairs (the `upsert` and `clear` of a
|
|
todo's `op`), empty for other fields.
|
|
- `limits`: `{max_nodes, max_depth, max_flows}`.
|
|
|
|
A controller that predates `schema` leaves it out of the answer; a client reads
|
|
it as optional.
|
|
|
|
**Requesting a review.** A `request_reviewer` node has one field, `reviewer`,
|
|
a forge login. For the PR the event concerns (`repo` and `number`; an event
|
|
about an issue sends nothing), the router sends a NATS request on
|
|
`$SWARM.router.sink.request-reviewer` with `{repo, number, reviewer}` and
|
|
swarm-controller answers with one line:
|
|
|
|
| Reply | Meaning |
|
|
| --------------- | ---------------------------------------------------------------------------------- |
|
|
| `requested` | the forge accepted the request |
|
|
| `skipped:<why>` | the reviewer's review of the PR's head already stands; the controller sent nothing |
|
|
| `error:<msg>` | the forge refused or didn't answer, or the request didn't decode |
|
|
|
|
The controller asks the forge under its own token, which the router never
|
|
holds. It skips a reviewer whose latest review is neither stale (the PR's head
|
|
moved past the commit it reviewed) nor dismissed, because requesting a review
|
|
from someone who already submitted one makes the forge dismiss it, approval
|
|
included. The controller asks again once the review is older than the head.
|
|
`hive-forge pr assign-reviewer` applies the same rule (`hive_types::review`).
|
|
|
|
The router sends the request once, after it queued the event's todo ops and
|
|
wrote the snapshot, and logs the answer. A controller that's down or answers
|
|
`error:` doesn't hold up the event or any agent's todos, and the request is
|
|
not retried: the next push asks again. A request repeated for the same head is
|
|
harmless: a reviewer with a pending request is a no-op for the forge, and one
|
|
who has reviewed gets `skipped`.
|
|
|
|
`head-classifier` requests the review classifier's review on a push to any open
|
|
PR. The classifier then runs as it does for any requested review
|
|
([`review-classifier.md`](review-classifier.md#trigger-and-lifecycle)), through
|
|
its own request queue, which replaces a waiting request for the same PR. A
|
|
repo the classifier's account can't read, or a PR that account opened, answers
|
|
`error:`.
|
|
|
|
**What the router runs.** It reads the key at most every 10 seconds, when an
|
|
event arrives, and runs the built-ins you left on, then the custom flows that
|
|
are on and pass the tree check. It logs a custom flow that fails the check,
|
|
leaves it out, and runs the rest.
|
|
|
|
| the key | the router runs |
|
|
| -------------------------------------- | ----------------------- |
|
|
| absent, deleted, or its bucket missing | the built-in flows |
|
|
| a document it reads | the above |
|
|
| a document it can't read | the flows it ran before |
|
|
| a read that fails | the flows it ran before |
|
|
|
|
"Can't read" means not JSON, not a flows document, or a `version` other than 1.
|
|
Before the router has read any document, "before" means the built-ins, so a bad
|
|
document never leaves it with no flows. The controller refuses to overwrite a
|
|
document of a version it doesn't know.
|
|
|
|
**Deploying in any order.** The controller, the router and the UI can be on
|
|
different hosts. A router that predates `request_reviewer` reads a document
|
|
holding such a node (a custom flow with one) as one it can't read and keeps the
|
|
flows it had; it also has no `head-classifier`. A router that has it, with a
|
|
controller that predates it, gets no answer to the request and logs that. A controller that predates this leaves the key absent, and the
|
|
router runs the built-ins as before. A router that predates this ignores the
|
|
document, so the page and the API show a state the router isn't yet running.
|
|
|
|
## Deliveries and dry runs
|
|
|
|
**The deliveries log.** The router appends one row to the `NOTIFY_ROUTER_LOG`
|
|
stream for each delivery it makes, and the stream keeps the newest 500. A
|
|
delivery is a todo op queued for an agent or a review requested from the
|
|
controller, whether it went through or not. A row has:
|
|
|
|
| Field | Holds |
|
|
| ------------------------- | ---------------------------------------------------------------------------------- |
|
|
| `at` | Unix seconds |
|
|
| `flow`, `node` | the flow and the node that delivered |
|
|
| `event`, `repo`, `number` | the forge event type and the issue or PR it concerns |
|
|
| `kind`, `target` | `todo` and the agent, or `request_reviewer` and the reviewer |
|
|
| `ok`, `detail` | whether it went through, and the op and key, the controller's answer, or the error |
|
|
|
|
A review the controller skipped (`skipped: <why>`) is `ok`: nothing was
|
|
wrong. An event the router retries after a failed step writes its rows again,
|
|
so a retry shows twice, and the page says so under the table. A row that
|
|
takes longer than 2 seconds to write counts as failed: the router logs a
|
|
warning and the delivery goes on.
|
|
|
|
**The dry run.** The router runs a flows document over the newest 50 events of
|
|
the `SWARM_EVENTS` stream and returns, for each flow the document turns on,
|
|
how many of those events reached each node. A `match` node counts the events
|
|
that passed it. It uses the walk the router delivers with and delivers nothing:
|
|
no todo op, no review request, no log row and no snapshot written. It keeps
|
|
its own per-issue snapshots, starting empty, so the first assign event it
|
|
replays for an issue counts every assignee as newly added. Every recipient
|
|
counts as an agent.
|
|
|
|
**The API.** The router answers two NATS requests from swarm-controller, which
|
|
serves them next to `GET /api/notify/flows`:
|
|
|
|
| Route | Router subject | Answers |
|
|
| ---------------------------- | ------------------------------ | -------------------------------------------------------------------------------------------- |
|
|
| `GET /api/notify/deliveries` | `$SWARM.router.api.deliveries` | the newest rows, newest first; `?limit=` up to 500, 100 by default |
|
|
| `GET /api/notify/dry-run` | `$SWARM.router.api.dry-run` | the saved flows document, run over the newest 50 events |
|
|
| `POST /api/notify/dry-run` | `$SWARM.router.api.dry-run` | the body's draft flows document, checked as a `PUT` checks it, run over the newest 50 events |
|
|
|
|
The dry-run request carries a whole document. The `GET` sends the saved one;
|
|
the `POST` takes a `PUT` body (it ignores a `revision`), answers 400
|
|
with the `PUT` error for a document that fails the check, and forwards a valid one.
|
|
[`/notifications`](ui.md#notification-flows) shows both: the recent deliveries,
|
|
and the counts beside each node.
|
|
|
|
**Deploying in any order.** A router that predates these subjects doesn't
|
|
subscribe to them, and the queue answers the controller's request at once with
|
|
no responders. Both routes then answer 503 saying the router doesn't answer
|
|
and may be an older build, and the page shows that in place of the table and
|
|
the counts. A request the router hears and doesn't answer within 10 seconds is
|
|
the same 503. A router that answers with a refusal, or with something the
|
|
controller can't read, is 502. A controller that predates the routes has no
|
|
route: the page, if newer, shows the failure.
|
|
|
|
## Credentials and grants
|
|
|
|
| Principal | Grant |
|
|
| --------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| `swarm-notify-router`, in its container | store: `read` on `swarm/services/<router client id>/oidc/client`. Queue: its own consumer on `SWARM_EVENTS` (create, pull, ack) and `STREAM.MSG.GET` on it for the dry run, publish on `$SWARM.router.todo.*` and on `$SWARM.router.sink.request-reviewer`, create `ROUTER_TODOS`, create `NOTIFY_ROUTER_LOG`, publish its one subject and read it by sequence, its own `notify-router-state` bucket, read on the `notify-flows` bucket, `STREAM.INFO` on the agent-status bucket, which lists the agents, and subscribe to the two `$SWARM.router.api.*` request subjects and answer each once, for 10 seconds |
|
|
| swarm-controller | publish on `$SWARM.events.forge.*` and on the two `$SWARM.router.api.*` request subjects; subscribe to `$SWARM.router.sink.request-reviewer` and answer each request once, for 30 seconds; creates and writes the `notify-flows` bucket (every bucket, by its `$KV.*.>` grant) |
|
|
| an agent, under its own credential | create its own consumer `todo-<agent>` in the form whose filter is `$SWARM.router.todo.<agent>`, pull from it, and ack what it delivered |
|
|
|
|
No other client may publish on `$SWARM.router.todo.*`, `$SWARM.router.log`, the two
|
|
`$SWARM.router.api.*` subjects, `$SWARM.router.sink.request-reviewer` or
|
|
`$SWARM.events.forge.*`, so each has one writer, and the controller alone
|
|
hears the review requests and the router alone hears the api requests. The review request needs no secret of its own: the
|
|
controller asks the forge under the token it already holds. The agent's create subject
|
|
carries the consumer's filter, and the queue server checks it against the
|
|
request, so an agent can't create a consumer that delivers another agent's
|
|
todos.
|