hyperhive/docs/swarm/notify-router.md

Notification router

The swarm's swarm-notify-router service turns forge events into todos on the agents they concern. No hive is in the path, and the agent can be on any hive: swarm-controller keeps the events, the router decides, and each agent's harness takes its own todos off the queue.

Where it runs and how it's enabled: services.md.

It runs beside hive-forge-notify, which still delivers forge notifications the way it did. An agent can get the same news from both.

What an agent gets

Flow	Event	The agent's todo
assigned	someone assigns an agent to an issue or PR	you are assigned to issue <repo>#<n>: <title> (<url>), key forge:<repo>#<n>:assignment
unassigned	someone takes an agent off an issue or PR	the same key rewritten: you are not assigned to issue <repo>#<n> anymore: <title> (<url>)
closed	someone closes or merges the issue or PR	the router clears every router todo it queued for it, then queues closed on each agent that held one: <kind> <repo>#<n> was closed: <title> (<url>), key forge:<repo>#<n>:closed

The todos are subsystem router in the agent's todo list. A todo under the same key replaces the one before, so an issue gives an agent one assignment todo however often it changes hands. A new event reopens a todo the agent marked done.

The closed todo has its own key, apart from the assignment todo. A reopened issue leaves it in place until the next close replaces it; a new assignment writes the assignment key and leaves it alone.

Only agents get todos. An assignee with no agent of that name gets nothing. An agent assigned before it first reports its status gets its todo on the issue's next assign event.

Forgejo's assign events carry the assignees after the change, not which one changed. The router keeps the assignees it saw on the last assign event per issue and takes the difference. With no earlier assign event for an issue, every current assignee counts as newly assigned and nobody as removed: an router notices an unassign only on an issue it has seen an assign event for.

The path

1. swarm-controller registers two instance-wide hooks, one for the issues and issue_assign events at /webhook/forge/issues and one for pull_request, pull_request_assign and pull_request_sync (a push to an open PR's branch) at /webhook/forge/pull-request. At each start it adds any of these events a registered hook lacks.
2. On each delivery the controller checks the HMAC and publishes the body, unchanged, to the SWARM_EVENTS stream on $SWARM.events.forge.<X-Forgejo-Event-Type>, such as $SWARM.events.forge.issue_assign. The stream keeps a delivery 7 days and drops a redelivery by its X-Forgejo-Delivery id. A delivery the stream did not keep answers 503, so the forge's delivery log shows it.
3. The router drains its durable consumer, notify-router, in order. For each event it runs every flow, queues the todo ops the flows emit, writes the issue's snapshot to its notify-router-state bucket, then acks the event. A failed step stops the batch: that event and the rest come back in stream order after 30 seconds, and the stream drops the ops queued twice by their message id.
4. Each op goes to the ROUTER_TODOS stream on the agent's own subject, $SWARM.router.todo.<agent>. Work-queue retention: an op stays until its agent acks it, up to 30 days and 1000 per agent.
5. The agent's harness creates its own durable consumer, todo-<agent>, applies each op to its todo list and acks it. An agent that's down, or whose harness predates this, leaves its ops waiting.

The router holds no forge token and no webhook secret, and registers no hook.

Flows

A flow is a graph of nodes joined by edges: one source node naming the forge event types it starts from, match nodes that pass when every condition holds, todo nodes that emit one todo op per recipient, and request_reviewer nodes that ask for a review. A match node can have more than one child, and each one runs, so one event can reach several leaves. A flow must be a tree: one source, one parent for every other node, every node reachable, no children under a todo or a request_reviewer. The most is 64 nodes, 16 levels deep, and 100 custom flows. A todo node sends to the agents an event field holds (assignees_added, todo_holders, …) or to one named agent.

The router ships the three flows above, each a source → match on action → todo, and head-classifier: source pull_request_sync → request_reviewer for review-classifier (below). The code defines them (swarm_wire::notify_flows::builtins), so a build that changes one reaches every swarm. Each switches off on /notifications. A custom flow has the same type and the same evaluator, so it can do anything a built-in does.

The stored document. swarm-controller keeps the rest in one document, the all key of the notify-flows bucket: the ids of the built-in flows you switched off, and the custom flows. Only the controller writes it, through GET and PUT /api/notify/flows. A PUT replaces the whole document in one compare-and-set. The controller answers 400 unless every flow passes the tree check above and reads only what exists: a source event from issues, issue_assign, pull_request, pull_request_assign or pull_request_sync, and condition and recipient fields from the event's field list. The same goes for the {field} placeholders in a todo's key and text. A PUT must carry the revision it read, 0 for no document, and gets 409 when the stored revision is another. Swarm UI's /notifications page shows the document and switches built-ins.

The schema. GET /api/notify/flows also returns a schema object, built from the constants the check reads, so an editor offers only what a PUT accepts:

• source_events: [{name, description}], the source events.
• fields: [{name, description, values, values_from?, closed}], the condition and recipient fields, and the {field} placeholders. values lists what the field holds, as far as the controller knows, as {name, description} pairs, such as the issue and PR of kind.
• node_kinds: [{kind, description, fields}] for source, match, todo and request_reviewer; each field is {name, description, required, values, values_from?, closed}, and values lists the accepted {name, description} pairs (the upsert and clear of a todo's op), empty for other fields.
• values_from on a field says where an editor asks for values that are only known live: agents is the names GET /api/agents lists, repos the ones GET /api/repos lists. closed says values and that source are every value the field takes, which an editor can show as a dropdown. When false, they are suggestions: the field still takes any other text, such as an action the forge adds later or the login of a reviewer who is no agent.
• limits: {max_nodes, max_depth, max_flows}.

A controller that predates schema leaves it out of the answer; a client reads it as optional.

Requesting a review. A request_reviewer node has one field, reviewer, a forge login. For the PR the event concerns (repo and number; an event about an issue sends nothing), the router sends a NATS request on $SWARM.router.sink.request-reviewer with {repo, number, reviewer} and swarm-controller answers with one line:

Reply	Meaning
requested	the forge accepted the request
skipped:<why>	the reviewer's review of the PR's head already stands; the controller sent nothing
error:<msg>	the forge refused or didn't answer, or the request didn't decode

The controller asks the forge under its own token, which the router never holds. It skips a reviewer whose latest review is neither stale (the PR's head moved past the commit it reviewed) nor dismissed, because requesting a review from someone who already submitted one makes the forge dismiss it, approval included. The controller asks again once the review is older than the head. hive-forge pr assign-reviewer applies the same rule (hive_types::review).

The router sends the request once, after it queued the event's todo ops and wrote the snapshot, and logs the answer. A controller that's down or answers error: doesn't hold up the event or any agent's todos, and the request is not retried: the next push asks again. A request repeated for the same head is harmless: a reviewer with a pending request is a no-op for the forge, and one who has reviewed gets skipped.

head-classifier requests the review classifier's review on a push to any open PR. The classifier then runs as it does for any requested review (review-classifier.md), through its own request queue, which replaces a waiting request for the same PR. A repo the classifier's account can't read, or a PR that account opened, answers error:.

What the router runs. It reads the key at most every 10 seconds, when an event arrives, and runs the built-ins you left on, then the custom flows that are on and pass the tree check. It logs a custom flow that fails the check, leaves it out, and runs the rest.

the key	the router runs
absent, deleted, or its bucket missing	the built-in flows
a document it reads	the above
a document it can't read	the flows it ran before
a read that fails	the flows it ran before

"Can't read" means not JSON, not a flows document, or a version other than 1. Before the router has read any document, "before" means the built-ins, so a bad document never leaves it with no flows. The controller refuses to overwrite a document of a version it doesn't know.

Deploying in any order. The controller, the router and the UI can be on different hosts. A router that predates request_reviewer reads a document holding such a node (a custom flow with one) as one it can't read and keeps the flows it had; it also has no head-classifier. A router that has it, with a controller that predates it, gets no answer to the request and logs that. A controller that predates this leaves the key absent, and the router runs the built-ins as before. A router that predates this ignores the document, so the page and the API show a state the router isn't yet running.

Deliveries and dry runs

The deliveries log. The router appends one row to the NOTIFY_ROUTER_LOG stream for each delivery it makes, and the stream keeps the newest 500. A delivery is a todo op queued for an agent or a review requested from the controller, whether it went through or not. A row has:

Field	Holds
at	Unix seconds
flow, node	the flow and the node that delivered
event, repo, number	the forge event type and the issue or PR it concerns
kind, target	todo and the agent, or request_reviewer and the reviewer
ok, detail	whether it went through, and the op and key, the controller's answer, or the error

A review the controller skipped (skipped: <why>) is ok: nothing was wrong. An event the router retries after a failed step writes its rows again, so a retry shows twice, and the page says so under the table. A row that takes longer than 2 seconds to write counts as failed: the router logs a warning and the delivery goes on.

The dry run. The router runs a flows document over the newest 50 events of the SWARM_EVENTS stream and returns, for each flow the document turns on, how many of those events reached each node. A match node counts the events that passed it. It uses the walk the router delivers with and delivers nothing: no todo op, no review request, no log row and no snapshot written. It keeps its own per-issue snapshots, starting empty, so the first assign event it replays for an issue counts every assignee as newly added. Every recipient counts as an agent.

The API. The router answers two NATS requests from swarm-controller, which serves them next to GET /api/notify/flows:

Route	Router subject	Answers
GET /api/notify/deliveries	$SWARM.router.api.deliveries	the newest rows, newest first; ?limit= up to 500, 100 by default
GET /api/notify/dry-run	$SWARM.router.api.dry-run	the saved flows document, run over the newest 50 events
POST /api/notify/dry-run	$SWARM.router.api.dry-run	the body's draft flows document, checked as a PUT checks it, run over the newest 50 events

The dry-run request carries a whole document. The GET sends the saved one; the POST takes a PUT body (it ignores a revision), answers 400 with the PUT error for a document that fails the check, and forwards a valid one. /notifications shows both: the recent deliveries, and the counts beside each node.

Deploying in any order. A router that predates these subjects doesn't subscribe to them, and the queue answers the controller's request at once with no responders. Both routes then answer 503 saying the router doesn't answer and may be an older build, and the page shows that in place of the table and the counts. A request the router hears and doesn't answer within 10 seconds is the same 503. A router that answers with a refusal, or with something the controller can't read, is 502. A controller that predates the routes has no route: the page, if newer, shows the failure.

Credentials and grants

Principal	Grant
swarm-notify-router, in its container	store: read on swarm/services/<router client id>/oidc/client. Queue: its own consumer on SWARM_EVENTS (create, pull, ack) and STREAM.MSG.GET on it for the dry run, publish on $SWARM.router.todo.* and on $SWARM.router.sink.request-reviewer, create ROUTER_TODOS, create NOTIFY_ROUTER_LOG, publish its one subject and read it by sequence, its own notify-router-state bucket, read on the notify-flows bucket, STREAM.INFO on the agent-status bucket, which lists the agents, and subscribe to the two $SWARM.router.api.* request subjects and answer each once, for 10 seconds
swarm-controller	publish on $SWARM.events.forge.* and on the two $SWARM.router.api.* request subjects; subscribe to $SWARM.router.sink.request-reviewer and answer each request once, for 30 seconds; creates and writes the notify-flows bucket (every bucket, by its $KV.*.> grant)
an agent, under its own credential	create its own consumer todo-<agent> in the form whose filter is $SWARM.router.todo.<agent>, pull from it, and ack what it delivered

No other client may publish on $SWARM.router.todo.*, $SWARM.router.log, the two $SWARM.router.api.* subjects, $SWARM.router.sink.request-reviewer or $SWARM.events.forge.*, so each has one writer, and the controller alone hears the review requests and the router alone hears the api requests. The review request needs no secret of its own: the controller asks the forge under the token it already holds. The agent's create subject carries the consumer's filter, and the queue server checks it against the request, so an agent can't create a consumer that delivers another agent's todos.