agents get a todo when assigned to an issue or PR, rewritten when unassigned and cleared on close
A new swarm service, swarm-notify-router, in its own container on the host
that sets deploy.notify-router.enable (defaulted on by allSwarmServices where
the host has the router's store leaf, like the claude login minter).
- swarm-controller registers instance-wide hooks for `issues` +
`issue_assign` and `pull_request` + `pull_request_assign`, updates the
event list of a hook already registered with fewer (leaving its active flag as it is), and keeps each
verified body unchanged in the SWARM_EVENTS stream on
$SWARM.events.forge.<X-Forgejo-Event-Type>, deduped by X-Forgejo-Delivery.
A delivery the stream did not keep answers 503.
- swarm-notify-router drains its durable consumer on that stream, runs its
flows (source -> match -> todo sink, stored as nodes plus edges and checked
to be a tree) and queues each op on $SWARM.router.todo.<agent> in the
work-queue stream ROUTER_TODOS. Built-in flows: assigned, unassigned (same
key, rewritten) and closed (prefix-clears every holder's router todos). A
per-issue snapshot in its notify-router-state bucket holds the last
assignees and the todo holders; only agents in the agent-status bucket get
ops, and an assignee that was not one is left out of the snapshot so the
next assign event retries it. After the first event that must be retried,
the rest of the batch is nakked so redelivery keeps stream order.
- hive-agent creates its own durable consumer todo-<agent> and applies each
op to its todo store as subsystem `router`, then acks; a refused op stops
the rest of its batch the same way. The store's (subsystem, key) dedup
makes a redelivery harmless. New Todos::clear_prefix.
- swarm-nats-auth: a --notify-router-client arm; the controller may publish
$SWARM.events.forge.*; each agent gets three {agent} templates (extended
consumer create whose filter the server checks, MSG.NEXT, ACK).
- Credentials: a notify-router store leaf minted by glue-bao-tls, a
swarm-notify-router bao policy (read on its own queue client secret), an
authelia machine client, the secret publisher entry, a journal forwarder
route. No manual step.
hive-forge-notify is unchanged, so agents also still get forge's own
notifications.
Refs #5571
This commit is contained in:
atlas
mara
parent
156dd541e6
commit
0f7b403803
53 changed files with 3587 additions and 96 deletions
|
|
@ -166,6 +166,10 @@ Swarm-level:
|
|||
- **`swarm-claude-login/`** — runs `claude auth login` for one agent and
|
||||
stores the credentials at `swarm/agents/<agent>/claude-credentials`.
|
||||
Design: [`docs/swarm/claude-login.md`](docs/swarm/claude-login.md).
|
||||
- **`swarm-notify-router/`** — forge events from swarm-controller's
|
||||
webhooks to todos on the agents they concern, through each agent's own
|
||||
consumer on the `ROUTER_TODOS` stream. Design:
|
||||
[`docs/swarm/notify-router.md`](docs/swarm/notify-router.md).
|
||||
- **`swarm-review-classifier/`** — one `claude --print` per PR
|
||||
classification, no tools, no forge credential. Design:
|
||||
[`docs/swarm/review-classifier.md`](docs/swarm/review-classifier.md).
|
||||
|
|
|
|||
19
Cargo.lock
generated
19
Cargo.lock
generated
|
|
@ -5028,6 +5028,23 @@ dependencies = [
|
|||
"tracing-subscriber",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "swarm-notify-router"
|
||||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"async-nats",
|
||||
"futures-util",
|
||||
"hive-log",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"swarm-queue-client",
|
||||
"swarm-secret-client",
|
||||
"swarm-wire",
|
||||
"tokio",
|
||||
"tracing",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "swarm-queue-client"
|
||||
version = "0.1.0"
|
||||
|
|
@ -5035,12 +5052,14 @@ dependencies = [
|
|||
"async-nats",
|
||||
"axum",
|
||||
"data-encoding",
|
||||
"futures-util",
|
||||
"hive-types",
|
||||
"reqwest",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"swarm-secret-client",
|
||||
"swarm-wire",
|
||||
"tempfile",
|
||||
"thiserror 2.0.18",
|
||||
"tokio",
|
||||
"tracing",
|
||||
|
|
|
|||
|
|
@ -36,6 +36,7 @@ members = [
|
|||
"swarm-matrix-client",
|
||||
"swarm-matrix-ctl",
|
||||
"swarm-nats-auth",
|
||||
"swarm-notify-router",
|
||||
"swarm-queue-client",
|
||||
"swarm-review-classifier",
|
||||
"swarm-logs",
|
||||
|
|
|
|||
|
|
@ -98,6 +98,8 @@ declarations.
|
|||
- **How do I log an agent in to claude from the swarm UI, and why doesn't a
|
||||
stale stored copy log it out again?** →
|
||||
[`swarm/claude-login.md`](swarm/claude-login.md).
|
||||
- **How does an agent get a todo when someone assigns it an issue, and lose it
|
||||
when the issue closes?** → [`swarm/notify-router.md`](swarm/notify-router.md).
|
||||
|
||||
## Scheduler, CI, observability
|
||||
|
||||
|
|
|
|||
|
|
@ -31,6 +31,9 @@ grafana, the controller, authelia, …) →
|
|||
- **review classifier** — planned, not implemented yet: checks a PR against
|
||||
its repo's own yes/no rules and posts the result as a review. →
|
||||
[`review-classifier.md`](review-classifier.md)
|
||||
- **notification router** — forge events turned into todos on the agents
|
||||
they concern: assigned, unassigned, cleared on close. →
|
||||
[`notify-router.md`](notify-router.md)
|
||||
- **swarm UI** — the operator's day-to-day surface, on the swarm apex,
|
||||
`admins` only. → [`ui.md`](ui.md)
|
||||
- **shared services** — one forge, homeserver, SSO, queue and metrics/logs
|
||||
|
|
@ -412,12 +415,18 @@ itself, all under `https://<swarm.domain>/webhook/forge/`:
|
|||
- an instance-wide `push` system hook, for commit metrics
|
||||
- an instance-wide `pull_request_review_request` system hook, which starts
|
||||
the [review classifier](review-classifier.md#trigger-and-lifecycle)
|
||||
- instance-wide `issues` + `issue_assign` and `pull_request` +
|
||||
`pull_request_assign` system hooks, whose deliveries it
|
||||
keeps in the `SWARM_EVENTS` stream for the
|
||||
[notification router](notify-router.md#the-path)
|
||||
|
||||
The controller **interprets** a delivery and sends hives a specific
|
||||
message — _the knowledge repo changed_, _deploy agent `foo` at rev
|
||||
`abc123`_ — rather than forwarding forge payloads for each hive to
|
||||
re-derive. Approval happens once, at the swarm level: a hive receives a
|
||||
decision, not an event to adjudicate.
|
||||
The `issues` and `pull_request` deliveries are the exception: no hive gets
|
||||
them, and the notification router, not the controller, decides what they mean.
|
||||
|
||||
A `config-pr` delivery reporting a PR merged into `main` queues a deploy
|
||||
of its `merge_commit_sha` on the one hive whose wanted state places the
|
||||
|
|
|
|||
|
|
@ -56,7 +56,7 @@ of the cell says how.
|
|||
<!-- vale write-good.Passive = NO -->
|
||||
|
||||
| store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull |
|
||||
| ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `swarm/agents/<agent>/matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again |
|
||||
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | ✅ a secondary that didn't come up at start, or whose token the homeserver rejected, is read from the store again two minutes later, then at doubling waits up to sixteen minutes, without restarting the daemon or its other accounts; one that's up and then rejected ends its sync loop, the daemon exits and its timer restarts it |
|
||||
| `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase for every account listed under the agent's `matrix/`, in a five-minute pass | the agent container itself, under the certificate its hive passed in, when `hive-matrix-daemon` brings the account up | ❌ never replaced: the account's server-side key backup opens only with the passphrase it was enabled with, so the pass writes only to an empty path | ❌ read when the account comes up; an account whose passphrase lands later turns backup on at the next daemon start |
|
||||
|
|
@ -74,7 +74,7 @@ of the cell says how.
|
|||
| `swarm/hives/<hive>/forge/token` | `swarm-controller`, with the forge's admin API, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `forge::reader_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ hive-c0re reads the store on every forge read |
|
||||
| `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated |
|
||||
| `swarm/hives/<hive>/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | hive-c0re, under the hive's own certificate, when it first connects to the queue | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read until the first connect succeeds, retried with a backoff of up to five minutes while it's missing; then held in memory and read again when authelia refuses it |
|
||||
| `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate — the queue responder's under `swarm-nats-auth`'s, the homeserver's under `swarm-matrix-ctl`'s, the forge's under `swarm-forge-publisher`'s, the review classifier's under `swarm-review-classifier`'s, the CI runner's, Grafana's, the log store's, the metrics store's, the forge's and the homeserver's journal forwarders' under `swarm-ci-runner`'s, `swarm-grafana-oidc`'s, `swarm-logs-auth`'s, `swarm-metrics-auth`'s, `swarm-forge-publisher`'s and `swarm-matrix-ctl`'s, authelia's journal forwarder's under `swarm-secret-publisher`'s; the bao UI's by the store's granter | must be stated | must be stated |
|
||||
| `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate — the queue responder's under `swarm-nats-auth`'s, the homeserver's under `swarm-matrix-ctl`'s, the forge's under `swarm-forge-publisher`'s, the review classifier's under `swarm-review-classifier`'s, the notification router's under `swarm-notify-router`'s, the CI runner's, Grafana's, the log store's, the metrics store's, the forge's and the homeserver's journal forwarders' under `swarm-ci-runner`'s, `swarm-grafana-oidc`'s, `swarm-logs-auth`'s, `swarm-metrics-auth`'s, `swarm-forge-publisher`'s and `swarm-matrix-ctl`'s, authelia's journal forwarder's under `swarm-secret-publisher`'s; the bao UI's by the store's granter | must be stated | must be stated |
|
||||
| `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge's admin API, at start and in a five-minute pass; its grant on the path is write-only | `swarm-bao-ci-runner` on the runner's host, under the runner's own certificate, into a file under the host's `/run` that only `swarm-ci-runner-register` in the runner's container loads; no other principal's policy reads it | ❌ the forge hands out one token until an admin resets it; the controller writes a reset token within five minutes | ✅ `swarm-bao-ci-runner` reads it at boot, on deploy and every ten minutes, and restarts the runner when the value changed |
|
||||
| `swarm/services/<grafana client id>/metrics/bearer` | `swarm-bao-grafana-metrics-bearer` on Grafana's host, under Grafana's certificate, when the store holds none | `swarm-metrics-auth` on the metrics store's host, under its own certificate, at most once a minute (every three seconds while the store fails), to check the bearer Grafana's datasource presents at the store's read API, and `swarm-logs-grafana-auth` on the log store's host, under `swarm-logs-auth`'s certificate, to check the same bearer at the log store's query API | ❌ nothing re-mints it; an operator deletes it and restarts `swarm-bao-grafana-metrics-bearer` | ✅ `swarm-metrics-auth` reads it again once its copy is a minute old. ❌ Grafana reads its file copy only as it starts; the bearer unit restarts Grafana when the copy changes |
|
||||
| `swarm/services/review-classifier/anthropic-api-key` | the operator, by hand in the store's UI, field `api_key` | `swarm-review-classifier` in its container, under its own certificate, at the start of every API key run; held for that run and handed to `claude`'s environment only; no other principal's policy reads it | ❌ an operator's key; replaced by writing a new one | ✅ read on every run, so a written or removed key takes effect on the next run |
|
||||
|
|
|
|||
87
docs/swarm/notify-router.md
Normal file
87
docs/swarm/notify-router.md
Normal file
|
|
@ -0,0 +1,87 @@
|
|||
# Notification router
|
||||
|
||||
The swarm's `swarm-notify-router` service turns forge events into todos on the
|
||||
agents they concern. No hive is in the path, and the agent can be on any hive:
|
||||
swarm-controller keeps the events, the router decides, and each agent's harness
|
||||
takes its own todos off the queue.
|
||||
|
||||
Where it runs and how it's enabled: [`services.md`](services.md#notification-router).
|
||||
|
||||
It runs beside `hive-forge-notify`, which still delivers forge notifications
|
||||
the way it did. An agent can get the same news from both.
|
||||
|
||||
## What an agent gets
|
||||
|
||||
| Flow | Event | The agent's todo |
|
||||
| ------------ | ------------------------------------------ | ------------------------------------------------------------------------------------------- |
|
||||
| `assigned` | someone assigns an agent to an issue or PR | `you are assigned to issue <repo>#<n>: <title> (<url>)`, key `forge:<repo>#<n>:assignment` |
|
||||
| `unassigned` | someone takes an agent off an issue or PR | the same key rewritten: `you are not assigned to issue <repo>#<n> anymore: <title> (<url>)` |
|
||||
| `closed` | someone closes or merges the issue or PR | the router clears every router todo it queued for it, on every agent that holds one |
|
||||
|
||||
The todos are subsystem `router` in the agent's todo list. A todo under the
|
||||
same key replaces the one before, so an issue gives an agent one assignment
|
||||
todo however often it changes hands. A new event reopens a todo the agent
|
||||
marked done.
|
||||
|
||||
Only agents get todos. An assignee with no agent of that name gets nothing.
|
||||
An agent assigned before it first reports its status gets its todo on the
|
||||
issue's next assign event.
|
||||
|
||||
Forgejo's assign events carry the assignees after the change, not which one
|
||||
changed. The router keeps the assignees it saw on the last assign event per
|
||||
issue and takes the difference. With no earlier assign event for an issue,
|
||||
every current assignee counts as newly assigned and nobody as removed: an
|
||||
router notices an unassign only on an issue it has seen an assign event for.
|
||||
|
||||
## The path
|
||||
|
||||
1. swarm-controller registers two instance-wide hooks, one for the `issues`
|
||||
and `issue_assign` events at `/webhook/forge/issues` and one for
|
||||
`pull_request` and `pull_request_assign` at `/webhook/forge/pull-request`.
|
||||
At each start it adds any of these events a registered hook lacks.
|
||||
2. On each delivery the controller checks the HMAC and publishes the body,
|
||||
unchanged, to the `SWARM_EVENTS` stream on
|
||||
`$SWARM.events.forge.<X-Forgejo-Event-Type>`, such as
|
||||
`$SWARM.events.forge.issue_assign`. The stream keeps a delivery 7 days and
|
||||
drops a redelivery by its `X-Forgejo-Delivery` id. A delivery the stream did
|
||||
not keep answers 503, so the forge's delivery log shows it.
|
||||
3. The router drains its durable consumer, `notify-router`, in order. For each
|
||||
event it runs every flow, queues the todo ops the flows emit, writes the
|
||||
issue's snapshot to its `notify-router-state` bucket, then acks the event. A
|
||||
failed step stops the batch: that event and the rest come back in stream
|
||||
order after 30 seconds, and the stream drops the ops queued twice by their
|
||||
message id.
|
||||
4. Each op goes to the `ROUTER_TODOS` stream on the agent's own subject,
|
||||
`$SWARM.router.todo.<agent>`. Work-queue retention: an op stays until its
|
||||
agent acks it, up to 30 days and 1000 per agent.
|
||||
5. The agent's harness creates its own durable consumer, `todo-<agent>`,
|
||||
applies each op to its todo list and acks it. An agent that's down, or
|
||||
whose harness predates this, leaves its ops waiting.
|
||||
|
||||
The router holds no forge token and no webhook secret, and registers no hook.
|
||||
|
||||
## Flows
|
||||
|
||||
A flow is a graph of nodes joined by edges: one `source` node naming the forge
|
||||
event types it starts from, `match` nodes that pass when every condition
|
||||
holds, and `sink` nodes that emit one todo op per recipient. A match node can
|
||||
have more than one child, and each one runs, so one event can reach several
|
||||
sinks. A flow must be a tree: one source, one parent for every other node,
|
||||
every node reachable, no children under a sink.
|
||||
|
||||
The router ships the three flows above, each a source → match on `action` →
|
||||
todo sink.
|
||||
|
||||
## Credentials and grants
|
||||
|
||||
| Principal | Grant |
|
||||
| --------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `swarm-notify-router`, in its container | store: `read` on `swarm/services/<router client id>/oidc/client`. Queue: its own consumer on `SWARM_EVENTS` (create, pull, ack), publish on `$SWARM.router.todo.*`, create `ROUTER_TODOS`, its own `notify-router-state` bucket, and `STREAM.INFO` on the agent-status bucket, which lists the agents |
|
||||
| swarm-controller | publish on `$SWARM.events.forge.*` |
|
||||
| an agent, under its own credential | create its own consumer `todo-<agent>` in the form whose filter is `$SWARM.router.todo.<agent>`, pull from it, and ack what it delivered |
|
||||
|
||||
No other client may publish on `$SWARM.router.todo.*` or
|
||||
`$SWARM.events.forge.*`, so each has one writer. The agent's create subject
|
||||
carries the consumer's filter, and the queue server checks it against the
|
||||
request, so an agent can't create a consumer that delivers another agent's
|
||||
todos.
|
||||
|
|
@ -412,7 +412,7 @@ next controller restart.
|
|||
### Per-principal identities
|
||||
|
||||
Bao matches a cert-auth role on the certificate's subject, so a certificate is an
|
||||
identity and sharing one merges the identities. Each of these ten leaves has
|
||||
identity and sharing one merges the identities. Each of these eleven leaves has
|
||||
a role and a policy of its own, and reaches one path, except the review
|
||||
classifier's, the CI runner's, the builder's, the CI run poller's, the claude
|
||||
login minter's, the forge host's and Grafana's, which reach more than one:
|
||||
|
|
@ -426,6 +426,7 @@ login minter's, the forge host's and Grafana's, which reach more than one:
|
|||
| `swarm-bao-ci-runner`, and the runner's journal forwarder in its container | `ciRunnerClientCertFile` / `ciRunnerClientKeyFile` | `swarm/services/ci-runner/registration-token` and `swarm/services/<runner forwarder client id>/oidc/client` |
|
||||
| `swarm-review-classifier`, in its container | `reviewClassifierClientCertFile` / `reviewClassifierClientKeyFile` | `swarm/services/review-classifier/anthropic-api-key`, `swarm/services/review-classifier/acp-provider` and `swarm/services/<classifier client id>/oidc/client` |
|
||||
| `swarm-claude-login`, in its container | `claudeLoginClientCertFile` / `claudeLoginClientKeyFile` | creates and updates `swarm/agents/+/claude-credentials`; reads `swarm/services/<minter client id>/oidc/client` |
|
||||
| `swarm-notify-router`, in its container | `notifyRouterClientCertFile` / `notifyRouterClientKeyFile` | `swarm/services/<router client id>/oidc/client` |
|
||||
| `swarm-builder`, in its container | `builderClientCertFile` / `builderClientKeyFile` | `swarm/services/builder/forge/token` and `swarm/services/<builder client id>/oidc/client` |
|
||||
| `swarm-ci-metrics` | `ciMetricsClientCertFile` / `ciMetricsClientKeyFile` | `swarm/services/ci-metrics/forge/token` and `swarm/services/<poller client id>/oidc/client` |
|
||||
| `swarm-bao-forge-oidc` and the forge's controller-token mint and its journal forwarder in its container | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token`; reads `swarm/services/<forge client id>/oidc/client` and `swarm/services/<forge forwarder client id>/oidc/client` |
|
||||
|
|
|
|||
|
|
@ -342,16 +342,16 @@ gateway either way.
|
|||
| `swarm.otel.port` | Only if something else on the services host already claims `4319`. |
|
||||
|
||||
The `swarm-bao`, `swarm-nats`, `swarm-builder`, `swarm-claude-login`,
|
||||
`swarm-review-classifier`, `swarm-ci-runner`, `swarm-grafana`,
|
||||
`swarm-notify-router`, `swarm-review-classifier`, `swarm-ci-runner`, `swarm-grafana`,
|
||||
`swarm-victorialogs`, `swarm-victoriametrics`, `swarm-authelia`, `hive-forge`
|
||||
and `hive-matrix` containers each run a collector of their own that forwards the container's
|
||||
journal here, on a route and receiver per service
|
||||
(`https://<swarm.otel.domain>/<container>/`), authenticated as an OIDC client
|
||||
of that service's. The queue, builder, login minter and classifier reuse the
|
||||
service's own client; the store, CI runner, Grafana, log store, metrics store,
|
||||
of that service's. The queue, builder, login minter, notification router and
|
||||
classifier reuse the service's own client; the store, CI runner, Grafana, log store, metrics store,
|
||||
authelia, forge and homeserver have a forwarder client of their own
|
||||
(`swarm.<service>.otel.clientId`). Their records carry `service.name` (`bao`,
|
||||
`nats`, `builder`, `claude-login`, `review-classifier`, `ci-runner`,
|
||||
`nats`, `builder`, `claude-login`, `notify-router`, `review-classifier`, `ci-runner`,
|
||||
`grafana`, `victorialogs`, `victoriametrics`, `authelia`, `forgejo`,
|
||||
`tuwunel`) and no `hive` label. The forwarder of all but the store's is
|
||||
`nix/container-modules/swarm-journal-forwarder.nix`.
|
||||
|
|
@ -473,3 +473,24 @@ secret store by name, under its own `claude-login` leaf, and reaches claude's
|
|||
login endpoints over the network. On a host without the store, that leaf is
|
||||
the one file to copy, as for the CI runner. The flow and the version rule are
|
||||
in [`claude-login.md`](claude-login.md).
|
||||
|
||||
### Notification router
|
||||
|
||||
The notification router, in a `swarm-notify-router` container on the host
|
||||
that sets `deploy.notify-router.enable`. `allSwarmServices` defaults it on
|
||||
wherever the host has the router's store identity, on the review classifier's
|
||||
terms. Run one: it handles each issue's events in order.
|
||||
|
||||
swarm-controller keeps every `issues` and `pull_request` delivery from its
|
||||
instance-wide hooks in the `SWARM_EVENTS` stream. The router reads that
|
||||
stream and queues todos for the agents an event concerns: an agent assigned to
|
||||
an issue or pull request gets a todo, one taken off it gets that todo rewritten
|
||||
to say so, and closing it clears them. Each agent's harness takes its todos
|
||||
from the `ROUTER_TODOS` stream, so an agent that's down gets them when it's
|
||||
back.
|
||||
|
||||
It needs neither the forge nor a hive on its host, and holds no forge token.
|
||||
It reaches the queue and the secret store by name, under its own
|
||||
`notify-router` leaf. On a host without the store, that leaf is the one file to
|
||||
copy, as for the CI runner. The flows and the delivery path are in
|
||||
[`notify-router.md`](notify-router.md).
|
||||
|
|
|
|||
|
|
@ -178,6 +178,9 @@
|
|||
services.hyperhive.deploy.claude-login.package =
|
||||
lib.mkDefault
|
||||
self.packages.${pkgs.stdenv.hostPlatform.system}.swarm-claude-login;
|
||||
services.hyperhive.deploy.notify-router.package =
|
||||
lib.mkDefault
|
||||
self.packages.${pkgs.stdenv.hostPlatform.system}.swarm-notify-router;
|
||||
services.hyperhive.deploy.authelia.bridgePackage =
|
||||
lib.mkDefault
|
||||
self.packages.${pkgs.stdenv.hostPlatform.system}.swarm-authelia-bridge;
|
||||
|
|
|
|||
|
|
@ -44,6 +44,8 @@ sha1 = "0.10"
|
|||
# `agent-schedules`, `agent-inference` and `operator-profiles`: the schedule
|
||||
# ticker and the inference poller each read their own key, and the prompt
|
||||
# renderer the operator list, with one direct get.
|
||||
# `router-todo`: the consumer that drains the notification router's todo ops
|
||||
# for this agent.
|
||||
# `kv`/`notices` name buckets and streams this harness opens neither end of,
|
||||
# and the terminal publisher is a plain core-subject publish.
|
||||
swarm-queue-client = { workspace = true, features = [
|
||||
|
|
@ -52,6 +54,7 @@ swarm-queue-client = { workspace = true, features = [
|
|||
"agent-permissions",
|
||||
"agents-md-initial",
|
||||
"operator-profiles",
|
||||
"router-todo",
|
||||
] }
|
||||
# The spelling of the token the queue connection presents.
|
||||
swarm-agent-verify.workspace = true
|
||||
|
|
|
|||
|
|
@ -28,6 +28,7 @@ mod plugins;
|
|||
mod prompt;
|
||||
mod reminder_timer;
|
||||
mod reminders;
|
||||
mod router_todos;
|
||||
mod schedule_timer;
|
||||
mod serve_common;
|
||||
mod sqlite_writer;
|
||||
|
|
@ -458,6 +459,9 @@ fn spawn_todo_socket(
|
|||
// Swarm-scheduled prompts fire as todos too, from the list the
|
||||
// swarm keeps for this agent.
|
||||
tokio::spawn(schedule_timer::run(store.clone(), todo_wake.clone()));
|
||||
// And so do the todos the swarm notification router queues for
|
||||
// this agent.
|
||||
tokio::spawn(router_todos::run(store.clone(), todo_wake.clone()));
|
||||
Some(store)
|
||||
}
|
||||
Err(e) => {
|
||||
|
|
|
|||
308
hive-agent/src/router_todos.rs
Normal file
308
hive-agent/src/router_todos.rs
Normal file
|
|
@ -0,0 +1,308 @@
|
|||
//! Todos the swarm notification router queues for this agent.
|
||||
//!
|
||||
//! The router publishes each change as a `swarm_wire::router_todo::RouterTodoOp`
|
||||
//! on this agent's subject of the `ROUTER_TODOS` work-queue stream. This
|
||||
//! task creates the agent's own durable consumer on it (`todo-<agent>`,
|
||||
//! `swarm_queue_client::router_todo::open_own`), pulls, applies each op to the
|
||||
//! todo store as subsystem `router`, and acks it, which removes it from the
|
||||
//! stream. Ops queued while the agent was down wait there, so nothing is lost
|
||||
//! to downtime.
|
||||
//!
|
||||
//! **Redelivery is harmless.** The store dedups on `(subsystem, key)`, so an
|
||||
//! op applied twice changes nothing the second time, except that an upsert
|
||||
//! reopens a todo the agent marked done in between (`reopen_if_acked`).
|
||||
//!
|
||||
//! **Own credential only.** The queue grants this agent's credential creating
|
||||
//! its own consumer (the extended create subject, whose filter the server
|
||||
//! checks), pulling from it and acking it, and nothing else of the stream.
|
||||
//!
|
||||
//! Until the router has created the stream, the consumer create fails; this
|
||||
//! task retries every [`RETRY`].
|
||||
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
|
||||
use futures_util::StreamExt as _;
|
||||
use swarm_wire::router_todo::{Op, RouterTodoOp, TODO_SUBSYSTEM};
|
||||
use tokio::sync::Notify;
|
||||
|
||||
use crate::todos::Todos;
|
||||
|
||||
/// How long one pull waits for ops.
|
||||
const PULL_WAIT: Duration = Duration::from_secs(30);
|
||||
|
||||
/// How many ops one pull asks for.
|
||||
const BATCH: usize = 32;
|
||||
|
||||
/// How long to wait after a failure before creating the consumer again.
|
||||
const RETRY: Duration = Duration::from_mins(1);
|
||||
|
||||
/// What applying one op came to.
|
||||
#[derive(Debug, PartialEq, Eq)]
|
||||
enum Applied {
|
||||
/// The list changed in a way the agent should be woken for.
|
||||
Wake,
|
||||
/// Nothing to wake for: unchanged, or a clear.
|
||||
Quiet,
|
||||
}
|
||||
|
||||
/// Apply one op to the store.
|
||||
fn apply(todos: &Todos, op: &RouterTodoOp) -> anyhow::Result<Applied> {
|
||||
match op.op {
|
||||
Op::Upsert => {
|
||||
let (_, changed) = todos.upsert(
|
||||
TODO_SUBSYSTEM,
|
||||
Some(&op.key),
|
||||
&op.summary,
|
||||
Some(&op.source),
|
||||
true,
|
||||
)?;
|
||||
Ok(if changed {
|
||||
Applied::Wake
|
||||
} else {
|
||||
Applied::Quiet
|
||||
})
|
||||
}
|
||||
Op::Clear if op.prefix => {
|
||||
todos.clear_prefix(TODO_SUBSYSTEM, &op.key)?;
|
||||
Ok(Applied::Quiet)
|
||||
}
|
||||
Op::Clear => {
|
||||
todos.clear(TODO_SUBSYSTEM, Some(&op.key))?;
|
||||
Ok(Applied::Quiet)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Drain this agent's consumer for the harness's lifetime, if it has a label
|
||||
/// to name it with.
|
||||
pub async fn run(todos: Arc<Todos>, wake: Arc<Notify>) {
|
||||
let agent = crate::identity::label();
|
||||
if agent.is_empty() {
|
||||
tracing::warn!("this agent has no label; not taking router todos");
|
||||
return;
|
||||
}
|
||||
let client = crate::swarm_queue::client().await;
|
||||
loop {
|
||||
if let Err(e) = drain(&client, &agent, &todos, &wake).await {
|
||||
tracing::info!(
|
||||
error = format!("{e:#}"),
|
||||
"router todos: not draining; retrying"
|
||||
);
|
||||
}
|
||||
tokio::time::sleep(RETRY).await;
|
||||
}
|
||||
}
|
||||
|
||||
/// Create the consumer and pull from it until something fails.
|
||||
async fn drain(
|
||||
client: &async_nats::Client,
|
||||
agent: &str,
|
||||
todos: &Todos,
|
||||
wake: &Notify,
|
||||
) -> anyhow::Result<()> {
|
||||
swarm_queue_client::ensure_connected(client)?;
|
||||
let consumer = swarm_queue_client::router_todo::open_own(client, agent)
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!("creating this agent's consumer: {e}"))?;
|
||||
tracing::info!(%agent, "router todos: draining this agent's queue");
|
||||
loop {
|
||||
let mut batch = consumer
|
||||
.batch()
|
||||
.max_messages(BATCH)
|
||||
.expires(PULL_WAIT)
|
||||
.messages()
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!("requesting a batch: {e}"))?;
|
||||
let mut refused = false;
|
||||
while let Some(message) = batch.next().await {
|
||||
let message = message.map_err(|e| anyhow::anyhow!("pulling: {e}"))?;
|
||||
let ack = settle_op(&message.payload, &mut refused, |op| apply(todos, op), wake);
|
||||
message
|
||||
.ack_with(ack)
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!("acking: {e}"))?;
|
||||
}
|
||||
if refused {
|
||||
anyhow::bail!(
|
||||
"the store refused an op; it and the rest of its batch come back in order"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// How one pulled op is acked. Once the store has refused an op of the batch
|
||||
/// (`refused`), no later op is applied: they are all nakked for immediate
|
||||
/// redelivery, so two ops on one key are never applied out of order, and the
|
||||
/// caller waits before it pulls again.
|
||||
fn settle_op(
|
||||
payload: &[u8],
|
||||
refused: &mut bool,
|
||||
apply: impl FnOnce(&RouterTodoOp) -> anyhow::Result<Applied>,
|
||||
wake: &Notify,
|
||||
) -> async_nats::jetstream::AckKind {
|
||||
use async_nats::jetstream::AckKind;
|
||||
|
||||
if *refused {
|
||||
return AckKind::Nak(None);
|
||||
}
|
||||
let op = match serde_json::from_slice::<RouterTodoOp>(payload) {
|
||||
Ok(op) => op,
|
||||
Err(e) => {
|
||||
tracing::warn!(error = %e, "router todos: an undecodable op; dropped");
|
||||
return AckKind::Term;
|
||||
}
|
||||
};
|
||||
match apply(&op) {
|
||||
Ok(applied) => {
|
||||
tracing::info!(key = %op.key, op = ?op.op, ?applied, "router todos: applied");
|
||||
if applied == Applied::Wake {
|
||||
wake.notify_one();
|
||||
}
|
||||
AckKind::Ack
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::warn!(key = %op.key, error = ?e, "router todos: the store refused an op");
|
||||
*refused = true;
|
||||
AckKind::Nak(None)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn store() -> (tempfile::TempDir, Todos) {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let db = Todos::open(&dir.path().join("todos.sqlite")).unwrap();
|
||||
(dir, db)
|
||||
}
|
||||
|
||||
fn upsert(key: &str, summary: &str) -> RouterTodoOp {
|
||||
RouterTodoOp {
|
||||
op: Op::Upsert,
|
||||
key: key.to_owned(),
|
||||
prefix: false,
|
||||
summary: summary.to_owned(),
|
||||
source: "router:assigned".to_owned(),
|
||||
}
|
||||
}
|
||||
|
||||
fn clear(key: &str, prefix: bool) -> RouterTodoOp {
|
||||
RouterTodoOp {
|
||||
op: Op::Clear,
|
||||
key: key.to_owned(),
|
||||
prefix,
|
||||
summary: String::new(),
|
||||
source: "router:closed".to_owned(),
|
||||
}
|
||||
}
|
||||
|
||||
fn keys(todos: &Todos) -> Vec<String> {
|
||||
let mut k: Vec<String> = todos
|
||||
.list(Some(TODO_SUBSYSTEM))
|
||||
.unwrap()
|
||||
.into_iter()
|
||||
.filter_map(|t| t.subsystem_key)
|
||||
.collect();
|
||||
k.sort();
|
||||
k
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_upsert_wakes_once_and_its_redelivery_is_quiet() {
|
||||
let (_dir, s) = store();
|
||||
let op = upsert("forge:o/r#1:assignment", "you are assigned");
|
||||
assert_eq!(apply(&s, &op).unwrap(), Applied::Wake);
|
||||
assert_eq!(apply(&s, &op).unwrap(), Applied::Quiet);
|
||||
let rows = s.list(Some(TODO_SUBSYSTEM)).unwrap();
|
||||
assert_eq!(rows.len(), 1);
|
||||
assert_eq!(rows[0].source.as_deref(), Some("router:assigned"));
|
||||
// A new text under the same key rewrites the one row and wakes.
|
||||
assert_eq!(
|
||||
apply(&s, &upsert("forge:o/r#1:assignment", "not assigned")).unwrap(),
|
||||
Applied::Wake
|
||||
);
|
||||
assert_eq!(s.list(Some(TODO_SUBSYSTEM)).unwrap().len(), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_clear_removes_one_key_and_a_prefix_clear_every_match() {
|
||||
// Another issue whose number starts with the first one's.
|
||||
const OTHER: &str = "forge:o/r#12:assignment"; // lint:allow: a test key, not a tracker reference
|
||||
let (_dir, s) = store();
|
||||
for key in ["forge:o/r#1:assignment", "forge:o/r#1:review", OTHER] {
|
||||
apply(&s, &upsert(key, "x")).unwrap();
|
||||
}
|
||||
assert_eq!(
|
||||
apply(&s, &clear("forge:o/r#1:review", false)).unwrap(),
|
||||
Applied::Quiet
|
||||
);
|
||||
assert_eq!(keys(&s), [OTHER, "forge:o/r#1:assignment"]);
|
||||
// Not a prefix clear: the key must match whole.
|
||||
apply(&s, &clear("forge:o/r#1:", false)).unwrap();
|
||||
assert_eq!(keys(&s).len(), 2);
|
||||
apply(&s, &clear("forge:o/r#1:", true)).unwrap();
|
||||
assert_eq!(keys(&s), [OTHER]);
|
||||
}
|
||||
|
||||
/// Two ops on one key: the first is refused, so the second is not applied
|
||||
/// ahead of it, and both are nakked to come back in order.
|
||||
#[test]
|
||||
fn after_a_refused_op_no_later_op_is_applied() {
|
||||
use async_nats::jetstream::AckKind;
|
||||
|
||||
let wake = Notify::new();
|
||||
let assigned = serde_json::to_vec(&upsert("forge:o/r#1:assignment", "assigned")).unwrap();
|
||||
let unassigned =
|
||||
serde_json::to_vec(&upsert("forge:o/r#1:assignment", "not assigned")).unwrap();
|
||||
let mut refused = false;
|
||||
let mut applied = Vec::new();
|
||||
let mut step = |payload: &[u8], refuse: bool| {
|
||||
settle_op(
|
||||
payload,
|
||||
&mut refused,
|
||||
|op| {
|
||||
applied.push(op.summary.clone());
|
||||
if refuse {
|
||||
anyhow::bail!("the store is locked")
|
||||
}
|
||||
Ok(Applied::Wake)
|
||||
},
|
||||
&wake,
|
||||
)
|
||||
};
|
||||
let first = step(&assigned, true);
|
||||
let second = step(&unassigned, false);
|
||||
assert_eq!(format!("{first:?}"), format!("{:?}", AckKind::Nak(None)));
|
||||
assert_eq!(format!("{second:?}"), format!("{:?}", AckKind::Nak(None)));
|
||||
assert_eq!(applied, ["assigned"]);
|
||||
assert!(refused);
|
||||
|
||||
// Control: with nothing refused, both apply and are acked.
|
||||
let mut refused = false;
|
||||
for payload in [&assigned, &unassigned] {
|
||||
let ack = settle_op(payload, &mut refused, |_| Ok(Applied::Quiet), &wake);
|
||||
assert_eq!(format!("{ack:?}"), format!("{:?}", AckKind::Ack));
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn router_ops_touch_no_other_subsystem() {
|
||||
let (_dir, s) = store();
|
||||
s.upsert(
|
||||
"forge",
|
||||
Some("forge:o/r#1:assignment"),
|
||||
"notify",
|
||||
None,
|
||||
false,
|
||||
)
|
||||
.unwrap();
|
||||
apply(&s, &upsert("forge:o/r#1:assignment", "router")).unwrap();
|
||||
apply(&s, &clear("forge:o/r#1:", true)).unwrap();
|
||||
let left = s.list(None).unwrap();
|
||||
assert_eq!(left.len(), 1);
|
||||
assert_eq!(left[0].subsystem, "forge");
|
||||
}
|
||||
}
|
||||
|
|
@ -212,6 +212,27 @@ impl Todos {
|
|||
Ok(n)
|
||||
}
|
||||
|
||||
/// Clear every todo of `subsystem` whose key starts with `prefix`, acked
|
||||
/// or not. Returns the number of rows deleted.
|
||||
///
|
||||
/// # Errors
|
||||
///
|
||||
/// Propagates the sqlite delete failure.
|
||||
///
|
||||
/// # Panics
|
||||
///
|
||||
/// Panics if the connection mutex is poisoned.
|
||||
pub fn clear_prefix(&self, subsystem: &str, prefix: &str) -> Result<usize> {
|
||||
let conn = self.conn.lock().unwrap();
|
||||
// `substr`, not `LIKE`: a key holds `_` and `%` literally.
|
||||
let n = conn.execute(
|
||||
"DELETE FROM todos WHERE subsystem = ?1 \
|
||||
AND substr(subsystem_key, 1, length(?2)) = ?2",
|
||||
params![subsystem, prefix],
|
||||
)?;
|
||||
Ok(n)
|
||||
}
|
||||
|
||||
/// Clear every todo `subsystem` owns — used by a producer that rebuilds
|
||||
/// its whole set on restart (cancel-and-recreate). Returns the number
|
||||
/// of rows deleted.
|
||||
|
|
@ -392,6 +413,34 @@ mod tests {
|
|||
(dir, db)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn clear_prefix_removes_matching_keys_of_one_subsystem_acked_or_not() {
|
||||
let (_dir, s) = store();
|
||||
let (acked, _) = s
|
||||
.upsert("router", Some("forge:o/r#5:assignment"), "a", None, true)
|
||||
.unwrap();
|
||||
s.mark_done(acked).unwrap();
|
||||
s.upsert("router", Some("forge:o/r#5:review"), "b", None, true)
|
||||
.unwrap();
|
||||
s.upsert("router", Some("forge:o/r#55:assignment"), "c", None, true) // lint:allow: test key, not a tracker reference
|
||||
.unwrap();
|
||||
s.upsert("forge", Some("forge:o/r#5:x"), "d", None, true)
|
||||
.unwrap();
|
||||
s.upsert("router", Some("forge:axb#5:x"), "e", None, true)
|
||||
.unwrap();
|
||||
assert_eq!(s.clear_prefix("router", "forge:o/r#5:").unwrap(), 2);
|
||||
// `_` is no wildcard.
|
||||
assert_eq!(s.clear_prefix("router", "forge:a_b#5:").unwrap(), 0);
|
||||
let mut left: Vec<String> = s
|
||||
.list(None)
|
||||
.unwrap()
|
||||
.into_iter()
|
||||
.map(|t| t.summary)
|
||||
.collect();
|
||||
left.sort();
|
||||
assert_eq!(left, ["c", "d", "e"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn keyed_upsert_dedups_and_reports_changed() {
|
||||
let (_dir, s) = store();
|
||||
|
|
|
|||
|
|
@ -52,6 +52,7 @@
|
|||
./glue-ci-metrics-otel-oidc-client.nix
|
||||
./glue-ci-runner-otel-oidc-client.nix
|
||||
./glue-claude-login-oidc-client.nix
|
||||
./glue-notify-router-oidc-client.nix
|
||||
./glue-forge-otel-oidc-client.nix
|
||||
./glue-grafana-otel-oidc-client.nix
|
||||
./glue-matrix-otel-oidc-client.nix
|
||||
|
|
@ -68,6 +69,7 @@
|
|||
./swarm-bao.nix
|
||||
./swarm-builder.nix
|
||||
./swarm-claude-login.nix
|
||||
./swarm-notify-router.nix
|
||||
./swarm-ci-metrics.nix
|
||||
./swarm-ci-runner.nix
|
||||
./swarm-review-classifier.nix
|
||||
|
|
|
|||
|
|
@ -406,6 +406,25 @@ in
|
|||
'';
|
||||
};
|
||||
|
||||
notify-router.enable = lib.mkOption {
|
||||
type = lib.types.bool;
|
||||
default = false;
|
||||
example = true;
|
||||
description = ''
|
||||
Run the notification router in a `swarm-notify-router` container on
|
||||
this host. It reads the forge events swarm-controller keeps from its
|
||||
webhooks and queues todos for the agents they concern: assigned,
|
||||
unassigned, and cleared when the issue or pull request closes. Run
|
||||
it on one host: it handles each issue's events in order.
|
||||
|
||||
On by default where
|
||||
{option}`services.hyperhive.deploy.allSwarmServices` is set and this
|
||||
host has the router's store identity (the pair
|
||||
{option}`services.hyperhive.deploy.bao.notifyRouterClientCertFile`
|
||||
and `notifyRouterClientKeyFile`, minted on the store's host).
|
||||
'';
|
||||
};
|
||||
|
||||
builder.enable = lib.mkOption {
|
||||
type = lib.types.bool;
|
||||
default = false;
|
||||
|
|
|
|||
|
|
@ -121,6 +121,8 @@ in
|
|||
ciMetricsClientKeyFile = lib.mkDefault "${pkiDir}/ci-metrics-key.pem";
|
||||
claudeLoginClientCertFile = lib.mkDefault "${pkiDir}/claude-login.pem";
|
||||
claudeLoginClientKeyFile = lib.mkDefault "${pkiDir}/claude-login-key.pem";
|
||||
notifyRouterClientCertFile = lib.mkDefault "${pkiDir}/notify-router.pem";
|
||||
notifyRouterClientKeyFile = lib.mkDefault "${pkiDir}/notify-router-key.pem";
|
||||
forgePublisherClientCertFile = lib.mkDefault "${pkiDir}/forge-publisher.pem";
|
||||
forgePublisherClientKeyFile = lib.mkDefault "${pkiDir}/forge-publisher-key.pem";
|
||||
servicesIssuerClientCertFile = lib.mkDefault "${pkiDir}/services-issuer.pem";
|
||||
|
|
@ -326,6 +328,13 @@ in
|
|||
[ -s ${pkiDir}/claude-login.pem ] || ${signLeaf} ${pkiDir} claude-login \
|
||||
${lib.escapeShellArg deployCfg.bao.claudeLoginCommonName} "" clientAuth
|
||||
|
||||
# The notification router's, which reads its queue client secret.
|
||||
# Minted here for the matrix-ctl leaf's reason: the router is a swarm
|
||||
# service, so on a router host without the store this is the file an
|
||||
# operator copies.
|
||||
[ -s ${pkiDir}/notify-router.pem ] || ${signLeaf} ${pkiDir} notify-router \
|
||||
${lib.escapeShellArg deployCfg.bao.notifyRouterCommonName} "" clientAuth
|
||||
|
||||
# The forge's token publisher. Minted here for the matrix-ctl leaf's
|
||||
# reason: the forge is a swarm singleton, so on a forge host without the
|
||||
# store this is the file an operator copies.
|
||||
|
|
|
|||
39
nix/host-modules/glue-notify-router-oidc-client.nix
Normal file
39
nix/host-modules/glue-notify-router-oidc-client.nix
Normal file
|
|
@ -0,0 +1,39 @@
|
|||
# Glue: register the notification router as an OIDC client wherever authelia
|
||||
# runs.
|
||||
#
|
||||
# ONE PAIRING PER FILE — swarm-notify-router ← authelia, and nothing else.
|
||||
# Deleting this leaves a router whose token requests authelia answers with a
|
||||
# 401, and no minted secret for swarm-secret-publish to put in the store.
|
||||
#
|
||||
# ⚠️ Gated on authelia being HERE, and NOT on this host running the router,
|
||||
# for ./glue-controller-oidc-client.nix's reason: a client is a row in THIS
|
||||
# host's provider config. Registered whether or not any host enables the
|
||||
# router: nothing in `swarm.*` records that, so a swarm without one carries
|
||||
# one unused client and one unused minted secret.
|
||||
{
|
||||
lib,
|
||||
config,
|
||||
...
|
||||
}:
|
||||
let
|
||||
hyperhiveCfg = config.services.hyperhive;
|
||||
deployCfg = hyperhiveCfg.deploy;
|
||||
clientId = hyperhiveCfg.swarm.notify-router.clientId;
|
||||
in
|
||||
{
|
||||
config = lib.mkIf deployCfg.authelia.enable {
|
||||
services.hyperhive.swarm.authelia.oidc.clients = [
|
||||
{
|
||||
id = clientId;
|
||||
description = "HyperHive notification router";
|
||||
# `client_credentials`: a daemon authenticating as itself.
|
||||
kind = "machine";
|
||||
audience = [ clientId ];
|
||||
# The container's journal forwarder mints its token as this client, for
|
||||
# the swarm collector's receiver, which verifies it offline against
|
||||
# `/jwks.json`; authelia's default opaque handle fails there.
|
||||
accessTokenSignedResponseAlg = "RS256";
|
||||
}
|
||||
];
|
||||
};
|
||||
}
|
||||
|
|
@ -1148,6 +1148,16 @@ let
|
|||
}
|
||||
];
|
||||
|
||||
# The notification router: its own queue client secret, read at start by the
|
||||
# daemon inside its container, and nothing else.
|
||||
notifyRouterReaders = [
|
||||
{
|
||||
name = "swarm-notify-router";
|
||||
cn = baoDeploy.notifyRouterCommonName;
|
||||
policyText = readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.notify-router.clientId}/oidc/client";
|
||||
}
|
||||
];
|
||||
|
||||
# The forge host's one store identity. The forge's
|
||||
# `forgejo-swarm-controller-account`, in its container, writes the
|
||||
# controller's forge token under it, the one writer of that path and
|
||||
|
|
@ -2401,6 +2411,18 @@ in
|
|||
'';
|
||||
};
|
||||
|
||||
notifyRouterCommonName = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
default = "swarm-notify-router";
|
||||
description = ''
|
||||
Subject the store's `swarm-notify-router` cert-auth role accepts: the
|
||||
identity the notification router presents from inside its container.
|
||||
Its grant is read on its own queue client secret.
|
||||
|
||||
⚠️ Reserved as a hive name by ./swarm.nix, like its siblings.
|
||||
'';
|
||||
};
|
||||
|
||||
claudeLoginClientCertFile = lib.mkOption {
|
||||
type = lib.types.nullOr lib.types.str;
|
||||
default = null;
|
||||
|
|
@ -2426,6 +2448,31 @@ in
|
|||
'';
|
||||
};
|
||||
|
||||
notifyRouterClientCertFile = lib.mkOption {
|
||||
type = lib.types.nullOr lib.types.str;
|
||||
default = null;
|
||||
example = "/var/lib/swarm-bao-pki/notify-router.pem";
|
||||
description = ''
|
||||
Certificate the notification router presents to the store. Its subject
|
||||
must be {option}`services.hyperhive.deploy.bao.notifyRouterCommonName`.
|
||||
|
||||
./glue-bao-tls.nix points it at the leaf it mints on the store's host.
|
||||
On a router host without the store, it is a file the operator copies
|
||||
from the store's host. A path, never a value.
|
||||
'';
|
||||
};
|
||||
|
||||
notifyRouterClientKeyFile = lib.mkOption {
|
||||
type = lib.types.nullOr lib.types.str;
|
||||
default = null;
|
||||
example = "/var/lib/swarm-bao-pki/notify-router-key.pem";
|
||||
description = ''
|
||||
Private key for
|
||||
{option}`services.hyperhive.deploy.bao.notifyRouterClientCertFile`.
|
||||
A path, never a value.
|
||||
'';
|
||||
};
|
||||
|
||||
forgePublisherClientCertFile = lib.mkOption {
|
||||
type = lib.types.nullOr lib.types.str;
|
||||
default = null;
|
||||
|
|
@ -3524,6 +3571,8 @@ in
|
|||
|
||||
systemd.services.swarm-bao-claude-login-policy = readerPolicyUnit "write the claude login minter's credentials-writer bao policy and cert-auth role" claudeLoginWriters;
|
||||
|
||||
systemd.services.swarm-bao-notify-router-policy = readerPolicyUnit "write the notification router's client-secret-reader bao policy and cert-auth role" notifyRouterReaders;
|
||||
|
||||
systemd.services.swarm-bao-forge-publisher-policy = readerPolicyUnit "write the forge's controller-token-publisher bao policy and cert-auth role" forgePublisherWriters;
|
||||
|
||||
# A FOURTH sibling, same shape and same reasons as the two above. This
|
||||
|
|
|
|||
|
|
@ -867,6 +867,9 @@ in
|
|||
"--builder-client ${lib.escapeShellArg config.services.hyperhive.swarm.builder.clientId}"
|
||||
# Hears login steps and answers each by reply, and nothing else.
|
||||
"--claude-login-client ${lib.escapeShellArg config.services.hyperhive.swarm.claude-login.clientId}"
|
||||
# Drains its consumer on the forge events stream and queues
|
||||
# agents' todo ops (`swarm-nats-auth`'s `notify_router_permissions`).
|
||||
"--notify-router-client ${lib.escapeShellArg config.services.hyperhive.swarm.notify-router.clientId}"
|
||||
# What an agent may publish to, keyed on its own name: its
|
||||
# terminal rows and its turn-state header
|
||||
# (`hive-agent::swarm_agent_state`), two subject families so a
|
||||
|
|
@ -914,6 +917,16 @@ in
|
|||
# (`swarm_queue_client::agents_md_initial`), on the same terms: the
|
||||
# harness reads the text its `AGENTS.md` starts with.
|
||||
"--agent-token-publish-subject ${lib.escapeShellArg "\$\$JS.API.DIRECT.GET.KV_agents-md-initial.\$\$KV.agents-md-initial.{agent}"}"
|
||||
# Its own durable consumer on the notification router's
|
||||
# `ROUTER_TODOS` stream (`swarm_wire::router_todo`): create it
|
||||
# in the extended form whose last tokens are the filter, which
|
||||
# the server checks against the payload, so the consumer
|
||||
# delivers this agent's subject alone; pull from it; ack what
|
||||
# it delivered. No bare `CONSUMER.CREATE`, no `STREAM.INFO`
|
||||
# and no publish on the subjects: the router is their writer.
|
||||
"--agent-token-publish-subject ${lib.escapeShellArg "\$\$JS.API.CONSUMER.CREATE.ROUTER_TODOS.todo-{agent}.\$\$SWARM.router.todo.{agent}"}"
|
||||
"--agent-token-publish-subject ${lib.escapeShellArg "\$\$JS.API.CONSUMER.MSG.NEXT.ROUTER_TODOS.todo-{agent}"}"
|
||||
"--agent-token-publish-subject ${lib.escapeShellArg "\$\$JS.ACK.ROUTER_TODOS.todo-{agent}.>"}"
|
||||
# Every agent also reads the operator list, the one key of the
|
||||
# `operator-profiles` bucket, with one direct get. That grant
|
||||
# names no agent, so the responder adds it itself
|
||||
|
|
|
|||
235
nix/host-modules/swarm-notify-router.nix
Normal file
235
nix/host-modules/swarm-notify-router.nix
Normal file
|
|
@ -0,0 +1,235 @@
|
|||
# The swarm notification router, in a `swarm-notify-router` container on the
|
||||
# host that sets `deploy.notify-router.enable`, which `deploy.allSwarmServices`
|
||||
# defaults on where the host has the router's store identity. It reads the
|
||||
# forge events swarm-controller keeps from its webhooks and queues todo ops on
|
||||
# each agent's subject of `ROUTER_TODOS`, which the agent's harness applies.
|
||||
# docs/swarm/notify-router.md has the design.
|
||||
#
|
||||
# Its own network namespace on the bridge, as ./swarm-claude-login.nix has: it
|
||||
# reaches the swarm services by name and nothing on the host's loopback. It
|
||||
# talks only to the queue, the store and authelia, so it can run on any host.
|
||||
#
|
||||
# The container holds one credential, its own store leaf, which nspawn hands in
|
||||
# as a system credential with no copy on disk. The daemon reads its queue
|
||||
# client secret with it.
|
||||
{
|
||||
pkgs,
|
||||
lib,
|
||||
config,
|
||||
...
|
||||
}:
|
||||
let
|
||||
hyperhiveCfg = config.services.hyperhive;
|
||||
|
||||
cfg = hyperhiveCfg.deploy.notify-router;
|
||||
baoCfg = hyperhiveCfg.swarm.bao;
|
||||
baoDeploy = hyperhiveCfg.deploy.bao;
|
||||
natsCfg = hyperhiveCfg.swarm.nats;
|
||||
autheliaCfg = hyperhiveCfg.swarm.authelia;
|
||||
gatewayCfg = hyperhiveCfg.gateway;
|
||||
networkCfg = hyperhiveCfg.network;
|
||||
tlsCfg = hyperhiveCfg.deploy.hive-controller.tls;
|
||||
|
||||
machine = "swarm-notify-router";
|
||||
unit = "swarm-notify-router";
|
||||
clientId = hyperhiveCfg.swarm.notify-router.clientId;
|
||||
|
||||
# The container's journal forwarder pushes to its own receiver on the swarm
|
||||
# collector, the route `serviceProducers.swarm-notify-router` in
|
||||
# ./swarm-otel-service.nix declares, as ./swarm-claude-login.nix does.
|
||||
swarmOtelCfg = hyperhiveCfg.swarm.otel;
|
||||
gatewayPortSuffix = lib.optionalString (
|
||||
gatewayCfg.httpsPort != 443
|
||||
) ":${toString gatewayCfg.httpsPort}";
|
||||
forwarderEndpoint = "https://${swarmOtelCfg.domain}${gatewayPortSuffix}/${machine}";
|
||||
forwarderUnit = "swarm-journal-forwarder-oidc";
|
||||
# Loopback in the container's own netns, and distinct from the other service
|
||||
# containers' forwarders all the same.
|
||||
forwarderTelemetryPort = 8902;
|
||||
|
||||
# The daemon reaches authelia, the store and the queue by name over TLS, and
|
||||
# in self-signed mode their leaves chain to the hive CA.
|
||||
caTrust = import ./lib/hive-ca-trust.nix { inherit lib tlsCfg gatewayCfg; };
|
||||
caBundleModule = caTrust.trustBundle {
|
||||
inherit pkgs;
|
||||
name = machine;
|
||||
consumers = [
|
||||
unit
|
||||
# Go: `SSL_CERT_FILE` replaces its trust store, for the swarm
|
||||
# collector's and authelia's names.
|
||||
"opentelemetry-collector"
|
||||
];
|
||||
};
|
||||
|
||||
haveClientIdentity =
|
||||
baoDeploy.notifyRouterClientCertFile != null && baoDeploy.notifyRouterClientKeyFile != null;
|
||||
|
||||
# Credential id → host source, as ./swarm-claude-login.nix hands its leaf in.
|
||||
storeFiles = lib.filterAttrs (_: source: source != null) {
|
||||
notify-router-bao-cert = baoDeploy.notifyRouterClientCertFile;
|
||||
notify-router-bao-key = baoDeploy.notifyRouterClientKeyFile;
|
||||
notify-router-bao-server-ca = baoDeploy.serverCaFile;
|
||||
};
|
||||
# nixos-container expands `extraFlags` unquoted, so a source path must be
|
||||
# one word.
|
||||
oneWord = path: builtins.match ".*[[:space:]].*" path == null;
|
||||
credential = id: "/run/credentials/${unit}.service/${id}";
|
||||
forwarderCredential = id: "/run/credentials/${forwarderUnit}.service/${id}";
|
||||
|
||||
privateNetwork = true;
|
||||
in
|
||||
{
|
||||
options.services.hyperhive.swarm.notify-router.clientId = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
readOnly = true;
|
||||
default = "swarm-notify-router";
|
||||
description = ''
|
||||
The OAuth2 client id the notification router presents to the swarm
|
||||
queue. Read-only: the host running authelia registers it, the queue's
|
||||
auth-callout responder grants it the router's subjects, and the secret
|
||||
publisher carries its secret to the store, all from this one value.
|
||||
'';
|
||||
};
|
||||
|
||||
options.services.hyperhive.deploy.notify-router.package = lib.mkOption {
|
||||
type = lib.types.package;
|
||||
defaultText = lib.literalExpression "hyperhive.packages.\${system}.swarm-notify-router";
|
||||
description = "The notification router daemon package.";
|
||||
};
|
||||
|
||||
config = lib.mkIf cfg.enable {
|
||||
assertions = [
|
||||
{
|
||||
assertion = haveClientIdentity;
|
||||
message = ''
|
||||
services.hyperhive.deploy.notify-router.enable requires the router's
|
||||
swarm-secret-store client identity: set both
|
||||
|
||||
services.hyperhive.deploy.bao.notifyRouterClientCertFile
|
||||
services.hyperhive.deploy.bao.notifyRouterClientKeyFile
|
||||
|
||||
The router reads its queue client secret with it.
|
||||
|
||||
Both default to the pair glue-bao-tls.nix mints when
|
||||
services.hyperhive.deploy.bao.enable is true (the host that runs the
|
||||
store). Otherwise copy notify-router.pem and notify-router-key.pem
|
||||
from the store host's /var/lib/swarm-bao-pki/ and name them in those
|
||||
two options.
|
||||
'';
|
||||
}
|
||||
{
|
||||
assertion = lib.all oneWord (lib.attrValues storeFiles);
|
||||
message = ''
|
||||
The notification router's store identity paths must not contain
|
||||
whitespace: nspawn's --load-credential receives them through
|
||||
nixos-container's unquoted extra flags.
|
||||
'';
|
||||
}
|
||||
];
|
||||
|
||||
# The container hangs off the bridge, takes its address from the gateway's
|
||||
# DHCP and resolves through its dnsmasq.
|
||||
services.hyperhive.network.enable = lib.mkDefault true;
|
||||
services.hyperhive.gateway.enable = lib.mkDefault true;
|
||||
services.hyperhive.gateway.dns.enable = lib.mkDefault true;
|
||||
|
||||
# nspawn fails to start the container when a credential source is
|
||||
# missing, so where this host mints the leaf the container waits for it.
|
||||
systemd.services."container@${machine}" = lib.mkMerge [
|
||||
caTrust.containerOrdering
|
||||
{ after = [ "swarm-bao-pki.service" ]; }
|
||||
];
|
||||
|
||||
containers.${machine} = {
|
||||
autoStart = true;
|
||||
# The daemon keeps its state in the queue: its consumer's position and
|
||||
# its per-issue snapshots.
|
||||
ephemeral = false;
|
||||
# No `--link-journal=host`: the journal stays in the guest, where the
|
||||
# container's own forwarder reads it (../container-modules/swarm-journal-forwarder.nix).
|
||||
extraFlags = lib.mapAttrsToList (id: source: "--load-credential=${id}:${source}") storeFiles;
|
||||
inherit privateNetwork;
|
||||
hostBridge = networkCfg.bridgeName;
|
||||
bindMounts = caTrust.bindMount;
|
||||
|
||||
config =
|
||||
{ lib, ... }:
|
||||
{
|
||||
imports = [
|
||||
../container-modules/swarm-container.nix
|
||||
../container-modules/swarm-journal-forwarder.nix
|
||||
caBundleModule
|
||||
];
|
||||
|
||||
services.hyperhive.swarmContainer = {
|
||||
inherit privateNetwork;
|
||||
writesOwnResolvConf = false;
|
||||
};
|
||||
networking.nameservers = [ networkCfg.bridgeIp ];
|
||||
networking.interfaces.eth0.useDHCP = true;
|
||||
|
||||
users.users.${unit} = {
|
||||
isSystemUser = true;
|
||||
group = unit;
|
||||
};
|
||||
users.groups.${unit} = { };
|
||||
|
||||
# Authenticated as the daemon's own OIDC client, whose secret its
|
||||
# store identity already reads: one identity for the whole container.
|
||||
services.hyperhive.swarmJournalForwarder = {
|
||||
enable = true;
|
||||
serviceName = "notify-router";
|
||||
endpoint = forwarderEndpoint;
|
||||
inherit clientId;
|
||||
tokenUrl = "${autheliaCfg.url}/api/oidc/token";
|
||||
telemetryPort = forwarderTelemetryPort;
|
||||
store = {
|
||||
package = baoDeploy.package;
|
||||
addr = "https://${baoCfg.domain}:${toString baoCfg.port}";
|
||||
secretPath = "secret/swarm/services/${clientId}/oidc/client";
|
||||
clientCertFile = forwarderCredential "notify-router-bao-cert";
|
||||
clientKeyFile = forwarderCredential "notify-router-bao-key";
|
||||
caFile =
|
||||
if storeFiles ? notify-router-bao-server-ca then
|
||||
forwarderCredential "notify-router-bao-server-ca"
|
||||
else
|
||||
null;
|
||||
};
|
||||
};
|
||||
systemd.services.${forwarderUnit}.serviceConfig.LoadCredential = lib.attrNames storeFiles;
|
||||
|
||||
systemd.services.${unit} = {
|
||||
description = "route forge events to agents' todo lists";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
wants = [ "network-online.target" ];
|
||||
after = [ "network-online.target" ];
|
||||
environment = {
|
||||
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
|
||||
BAO_CLIENT_CERT = credential "notify-router-bao-cert";
|
||||
BAO_CLIENT_KEY = credential "notify-router-bao-key";
|
||||
SWARM_NOTIFY_ROUTER_NATS_URL = "tls://${natsCfg.domain}:${toString natsCfg.port}";
|
||||
SWARM_NOTIFY_ROUTER_OIDC_TOKEN_ENDPOINT = "${autheliaCfg.url}/api/oidc/token";
|
||||
SWARM_NOTIFY_ROUTER_OIDC_CLIENT_ID = clientId;
|
||||
}
|
||||
// lib.optionalAttrs (storeFiles ? notify-router-bao-server-ca) {
|
||||
BAO_CACERT = credential "notify-router-bao-server-ca";
|
||||
}
|
||||
// lib.optionalAttrs caTrust.useSelfSigned {
|
||||
SWARM_NOTIFY_ROUTER_OIDC_CA_FILE = caTrust.bundlePathFor machine;
|
||||
};
|
||||
serviceConfig = {
|
||||
ExecStart = "${cfg.package}/bin/swarm-notify-router";
|
||||
User = unit;
|
||||
Group = unit;
|
||||
UMask = "0077";
|
||||
# Bare ids: the system credentials nspawn handed in.
|
||||
LoadCredential = lib.attrNames storeFiles;
|
||||
Restart = "on-failure";
|
||||
RestartSec = "10s";
|
||||
SyslogIdentifier = unit;
|
||||
};
|
||||
};
|
||||
};
|
||||
};
|
||||
};
|
||||
}
|
||||
|
|
@ -391,9 +391,9 @@ in
|
|||
clientId = hyperhiveCfg.swarm.nats.clientId;
|
||||
port = lib.mkDefault 4392;
|
||||
};
|
||||
# The forwarders in ./swarm-builder.nix, ./swarm-claude-login.nix and
|
||||
# ./swarm-review-classifier.nix, which address their routes by container
|
||||
# name.
|
||||
# The forwarders in ./swarm-builder.nix, ./swarm-claude-login.nix,
|
||||
# ./swarm-notify-router.nix and ./swarm-review-classifier.nix, which
|
||||
# address their routes by container name.
|
||||
swarm-builder = {
|
||||
clientId = hyperhiveCfg.swarm.builder.clientId;
|
||||
port = lib.mkDefault 4393;
|
||||
|
|
@ -406,6 +406,10 @@ in
|
|||
clientId = hyperhiveCfg.swarm.review-classifier.clientId;
|
||||
port = lib.mkDefault 4395;
|
||||
};
|
||||
swarm-notify-router = {
|
||||
clientId = hyperhiveCfg.swarm.notify-router.clientId;
|
||||
port = lib.mkDefault 4404;
|
||||
};
|
||||
# The forwarders in ./swarm-ci-runner.nix, ./swarm-grafana.nix and
|
||||
# ./swarm-victorialogs.nix, each authenticated as a client of its own.
|
||||
swarm-ci-runner = {
|
||||
|
|
|
|||
|
|
@ -205,9 +205,10 @@ let
|
|||
# nixpkgs hardcodes `--link-journal=try-guest`, which keeps this container's
|
||||
# journal in the guest, so the host directory holds the host's units only.
|
||||
# Every container with a `serviceProducers` entry (`swarm-bao`, `swarm-nats`,
|
||||
# `swarm-builder`, `swarm-claude-login`, `swarm-review-classifier`,
|
||||
# `swarm-ci-runner`, `swarm-grafana`, `swarm-victorialogs`, `hive-forge`,
|
||||
# `hive-matrix`, `swarm-victoriametrics`, `swarm-authelia`) forwards its own
|
||||
# `swarm-builder`, `swarm-claude-login`, `swarm-notify-router`,
|
||||
# `swarm-review-classifier`, `swarm-ci-runner`, `swarm-grafana`,
|
||||
# `swarm-victorialogs`, `hive-forge`, `hive-matrix`, `swarm-victoriametrics`,
|
||||
# `swarm-authelia`) forwards its own
|
||||
# journal from its own collector, so this receiver never sees their units.
|
||||
hostJournalDir = "/var/log/journal";
|
||||
hostJournalMount = "/run/host-journal";
|
||||
|
|
|
|||
|
|
@ -137,4 +137,11 @@ in
|
|||
&& deployCfg.bao.claudeLoginClientCertFile != null
|
||||
&& deployCfg.bao.claudeLoginClientKeyFile != null
|
||||
);
|
||||
|
||||
# The notification router, defaulted on the same terms.
|
||||
config.services.hyperhive.deploy.notify-router.enable = lib.mkDefault (
|
||||
deployCfg.allSwarmServices
|
||||
&& deployCfg.bao.notifyRouterClientCertFile != null
|
||||
&& deployCfg.bao.notifyRouterClientKeyFile != null
|
||||
);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -58,6 +58,8 @@ let
|
|||
hyperhiveCfg.swarm.builder.clientId
|
||||
# The claude login minter, which reads it itself inside its container.
|
||||
hyperhiveCfg.swarm.claude-login.clientId
|
||||
# The notification router, which reads it itself inside its container.
|
||||
hyperhiveCfg.swarm.notify-router.clientId
|
||||
# The journal forwarders in the CI runner's, Grafana's, the log store's,
|
||||
# the forge's and the homeserver's containers, each reading it inside its
|
||||
# container.
|
||||
|
|
|
|||
|
|
@ -61,6 +61,7 @@ let
|
|||
deployCfg.bao.builderCommonName
|
||||
deployCfg.bao.ciMetricsCommonName
|
||||
deployCfg.bao.claudeLoginCommonName
|
||||
deployCfg.bao.notifyRouterCommonName
|
||||
deployCfg.bao.forgePublisherCommonName
|
||||
]
|
||||
# The per-hive subjects, spelled out per hive rather than as the prefix.
|
||||
|
|
|
|||
|
|
@ -193,6 +193,7 @@ let
|
|||
"swarm-bao-builder-policy"
|
||||
"swarm-bao-ci-metrics-policy"
|
||||
"swarm-bao-claude-login-policy"
|
||||
"swarm-bao-notify-router-policy"
|
||||
];
|
||||
|
||||
# Comment lines dropped first: both the HCL and the scripts explain
|
||||
|
|
@ -1834,6 +1835,32 @@ let
|
|||
in
|
||||
lib.hasInfix "/claude-login.pem ]" s && lib.hasInfix "swarm-claude-login \"\" clientAuth" s;
|
||||
}
|
||||
{
|
||||
# The router reads its own queue client secret and nothing else.
|
||||
# The claude login minter's grant is the control that `grantsIn` reads
|
||||
# a second path where there is one.
|
||||
name = "the notification router's grant is read on its own queue client secret alone";
|
||||
ok =
|
||||
let
|
||||
s = baoGrantHere.systemd.services.swarm-bao-notify-router-policy.script;
|
||||
grants = grantsIn s;
|
||||
minter = grantsIn baoGrantHere.systemd.services.swarm-bao-claude-login-policy.script;
|
||||
in
|
||||
(grantFor grants "secret/data/swarm/services/swarm-notify-router/oidc/client").caps == [ "read" ]
|
||||
&& lib.length grants == 1
|
||||
&& lib.length minter == 2
|
||||
&& lib.hasInfix "auth/cert/certs/swarm-notify-router" s
|
||||
&& lib.hasInfix "allowed_common_names=swarm-notify-router" s
|
||||
&& lib.hasInfix "token_policies=swarm-notify-router" s;
|
||||
}
|
||||
{
|
||||
name = "the PKI unit signs the notification router's leaf under its own subject";
|
||||
ok =
|
||||
let
|
||||
s = baoGrantHere.systemd.services.swarm-bao-pki.script;
|
||||
in
|
||||
lib.hasInfix "/notify-router.pem ]" s && lib.hasInfix "swarm-notify-router \"\" clientAuth" s;
|
||||
}
|
||||
{
|
||||
name = "the PKI unit signs the CI runner's leaf under its own subject";
|
||||
ok =
|
||||
|
|
|
|||
|
|
@ -76,7 +76,7 @@ let
|
|||
# The same swarm's IdP host, for the arm that the two agree.
|
||||
autheliaColocated = shared.autheliaOnly;
|
||||
|
||||
# The other three queue clients with a container of their own, each forwarding
|
||||
# The other four queue clients with a container of their own, each forwarding
|
||||
# its journal as the queue's does. One shared host: the switch that turns on
|
||||
# every swarm service, which other suites already evaluate.
|
||||
servicesHost = shared.swarmServicesWithController;
|
||||
|
|
@ -99,6 +99,12 @@ let
|
|||
credential = "review-classifier-bao";
|
||||
port = 8894;
|
||||
}
|
||||
{
|
||||
machine = "swarm-notify-router";
|
||||
name = "notify-router";
|
||||
credential = "notify-router-bao";
|
||||
port = 8902;
|
||||
}
|
||||
];
|
||||
forwarderFetchDir = "/run/credentials/swarm-journal-forwarder-oidc.service";
|
||||
|
||||
|
|
@ -483,7 +489,7 @@ let
|
|||
# The whole per-agent grant, compared as a list rather than by infix: a
|
||||
# wider subject added beside these (`$$JS.API.>`, another stream's name)
|
||||
# would pass every presence check above.
|
||||
name = "a verified agent's grant is exactly its own subjects, and its JetStream API subjects read its own schedule, inference and permissions keys";
|
||||
name = "a verified agent's grant is exactly its own subjects, and its JetStream API subjects read its own schedule, inference and permissions keys and drain its own router todo consumer";
|
||||
ok =
|
||||
let
|
||||
args = lib.splitString " " (responderOf natsQueueOnly).serviceConfig.ExecStart;
|
||||
|
|
@ -506,6 +512,9 @@ let
|
|||
"'$$JS.API.DIRECT.GET.KV_agent-inference.$$KV.agent-inference.{agent}'"
|
||||
"'$$JS.API.DIRECT.GET.KV_agent-permissions.$$KV.agent-permissions.{agent}'"
|
||||
"'$$JS.API.DIRECT.GET.KV_agents-md-initial.$$KV.agents-md-initial.{agent}'"
|
||||
"'$$JS.API.CONSUMER.CREATE.ROUTER_TODOS.todo-{agent}.$$SWARM.router.todo.{agent}'"
|
||||
"'$$JS.API.CONSUMER.MSG.NEXT.ROUTER_TODOS.todo-{agent}'"
|
||||
"'$$JS.ACK.ROUTER_TODOS.todo-{agent}.>'"
|
||||
];
|
||||
}
|
||||
{
|
||||
|
|
@ -776,10 +785,10 @@ let
|
|||
c != null && c.audience == [ "swarm-nats" ] && c.accessTokenSignedResponseAlg == "RS256";
|
||||
}
|
||||
{
|
||||
# The far end for the three: a receiver, authenticator, pipeline entry and
|
||||
# The far end for the four: a receiver, authenticator, pipeline entry and
|
||||
# gateway route each, checking that service's client as audience, beside
|
||||
# the queue's, on a collector host that runs none of them.
|
||||
name = "the swarm collector admits the builder, claude login and classifier forwarders on receivers of their own";
|
||||
name = "the swarm collector admits the builder, claude login, classifier and notification router forwarders on receivers of their own";
|
||||
ok =
|
||||
let
|
||||
h = shared.baoWithCollector;
|
||||
|
|
@ -797,13 +806,15 @@ let
|
|||
&& lib.elem "otlp/${svc.machine}" s.service.pipelines."logs/${otel.producerName}".receivers
|
||||
&& h.services.nginx.virtualHosts.${otel.domain}.locations ? "/${svc.machine}/"
|
||||
) forwarderServices
|
||||
# Four producers' receivers on the collector's host, none sharing a port.
|
||||
&& lib.length (lib.unique (ports ++ [ otel.serviceProducers.swarm-nats.port ])) == 4;
|
||||
# Their receivers and the queue's on the collector's host, none sharing a port.
|
||||
&&
|
||||
lib.length (lib.unique (ports ++ [ otel.serviceProducers.swarm-nats.port ]))
|
||||
== lib.length forwarderServices + 1;
|
||||
}
|
||||
{
|
||||
# The token each forwarder mints must name its own client as audience and
|
||||
# be a JWT, or the receiver refuses it.
|
||||
name = "the builder, claude login and classifier clients are registered with their own audience and a signed access token";
|
||||
name = "the builder, claude login, classifier and notification router clients are registered with their own audience and a signed access token";
|
||||
ok =
|
||||
let
|
||||
clients = shared.autheliaOnly.services.hyperhive.swarm.authelia.oidc.clients;
|
||||
|
|
|
|||
|
|
@ -279,6 +279,28 @@ let
|
|||
&& !(swarmServicesBaoElsewhere.containers ? swarm-claude-login)
|
||||
&& swarmServicesBaoElsewhere.services.hyperhive.deploy.nats.enable;
|
||||
}
|
||||
{
|
||||
# Same shape as the minter's: a static user in its own container. `nats`
|
||||
# is the control that the switch is on at all.
|
||||
name = "hosting the swarm's shared services runs the notification router in its container as a static user";
|
||||
ok =
|
||||
let
|
||||
c = swarmServicesHere.containers.swarm-notify-router.config;
|
||||
sc = c.systemd.services.swarm-notify-router.serviceConfig;
|
||||
in
|
||||
swarmServicesHere.services.hyperhive.deploy.notify-router.enable
|
||||
&& c.users.users.swarm-notify-router.isSystemUser
|
||||
&& sc.User == "swarm-notify-router"
|
||||
&& !(sc.DynamicUser or false)
|
||||
&& swarmServicesHere.services.hyperhive.deploy.nats.enable;
|
||||
}
|
||||
{
|
||||
name = "the notification router stays off where the shared services are hosted without the store";
|
||||
ok =
|
||||
!swarmServicesBaoElsewhere.services.hyperhive.deploy.notify-router.enable
|
||||
&& !(swarmServicesBaoElsewhere.containers ? swarm-notify-router)
|
||||
&& swarmServicesBaoElsewhere.services.hyperhive.deploy.nats.enable;
|
||||
}
|
||||
{
|
||||
# The CLI is the one the agents run, not the container's own nixpkgs'.
|
||||
# Unpinned is the control: it falls back to the container's own `claude-code`.
|
||||
|
|
@ -587,7 +609,7 @@ let
|
|||
# that lost a container can't pass by comparing fewer of them.
|
||||
name = "every service container restates its host-side privateNetwork";
|
||||
ok =
|
||||
lib.length (lib.attrNames serviceContainers) == 13
|
||||
lib.length (lib.attrNames serviceContainers) == 14
|
||||
&& lib.all (
|
||||
c: c.privateNetwork == (c.config.services.hyperhive.swarmContainer.privateNetwork or null)
|
||||
) (lib.attrValues serviceContainers);
|
||||
|
|
|
|||
|
|
@ -186,6 +186,7 @@ in
|
|||
swarm-builder = mkBinPackage "swarm-builder" "hyperhive swarm builder: agent config build checks";
|
||||
swarm-ci-metrics = mkBinPackage "swarm-ci-metrics" "hyperhive swarm CI run poller";
|
||||
swarm-claude-login = mkBinPackage "swarm-claude-login" "hyperhive claude login minter";
|
||||
swarm-notify-router = mkBinPackage "swarm-notify-router" "hyperhive swarm notification router";
|
||||
|
||||
# The only process allowed to write swarm-authelia's users database —
|
||||
# same "runs *inside* a container, not on the host" placement as
|
||||
|
|
|
|||
|
|
@ -107,11 +107,14 @@ swarm-controller-api = { workspace = true, features = ["schema"] }
|
|||
# config the crate spells.
|
||||
# `agent-term`: likewise every agent's terminal stream, plus the consumer it
|
||||
# replays that stream through.
|
||||
# `forge-events`: the stream it keeps verified forge deliveries in, for the
|
||||
# notification router.
|
||||
swarm-queue-client = { workspace = true, features = [
|
||||
"kv",
|
||||
"agent-permissions",
|
||||
"subagent-term",
|
||||
"agent-term",
|
||||
"forge-events",
|
||||
] }
|
||||
# `matrix_account.rs` writes the credential this daemon's route accepts. Same
|
||||
# crate the hive reads it back with, which is the point: the path, the field
|
||||
|
|
|
|||
|
|
@ -26,11 +26,11 @@ use forgejo_api::structs::{
|
|||
AddCollaboratorOption, AddCollaboratorOptionPermission, ChangeFileOperation,
|
||||
ChangeFileOperationOperation, ChangeFilesOptions, CommitStatusState,
|
||||
CreateBranchProtectionOption, CreateHookOption, CreateHookOptionConfig, CreateHookOptionType,
|
||||
CreateRepoOption, CreateStatusOption, CreateUserOption, EditUserOption, GetTreeQuery, GitEntry,
|
||||
IssueListIssuesQuery, IssueListIssuesQueryState, IssueListIssuesQueryType,
|
||||
RepoDownloadPullDiffOrPatchQuery, RepoGetContentsQuery, RepoGetPullRequestCommitsQuery,
|
||||
RepoGetPullRequestFilesQuery, RepoListPullRequestsQuery, RepoListPullRequestsQueryState,
|
||||
RepoSearchQuery, StateType,
|
||||
CreateRepoOption, CreateStatusOption, CreateUserOption, EditHookOption, EditUserOption,
|
||||
GetTreeQuery, GitEntry, IssueListIssuesQuery, IssueListIssuesQueryState,
|
||||
IssueListIssuesQueryType, RepoDownloadPullDiffOrPatchQuery, RepoGetContentsQuery,
|
||||
RepoGetPullRequestCommitsQuery, RepoGetPullRequestFilesQuery, RepoListPullRequestsQuery,
|
||||
RepoListPullRequestsQueryState, RepoSearchQuery, StateType,
|
||||
};
|
||||
use forgejo_api::{ApiErrorKind, ForgejoError};
|
||||
use futures_util::{StreamExt as _, TryStreamExt as _};
|
||||
|
|
@ -1614,29 +1614,8 @@ impl Client {
|
|||
async fn ensure_all_swarm_hooks(&self, public_base: &str, secret: &str) -> Result<()> {
|
||||
for kind in DeliveryKind::ALL {
|
||||
let target_url = kind.target_url(public_base);
|
||||
let (event, scope) = match kind {
|
||||
DeliveryKind::Knowledge => (
|
||||
"push",
|
||||
HookScope::Repo {
|
||||
org: KNOWLEDGE_ORG,
|
||||
repo: KNOWLEDGE_REPO,
|
||||
},
|
||||
),
|
||||
DeliveryKind::ConfigPr => ("pull_request", HookScope::Org { org: CONFIG_ORG }),
|
||||
// Instance-wide: commit/push activity is not scoped to one
|
||||
// org (unlike the two above) — it needs observing across
|
||||
// every repo the forge hosts, per the "no metric exists for
|
||||
// commits or pushes" tracker issue. Forgejo's
|
||||
// "global (system) webhook" (`admin_create_hook`) is exactly
|
||||
// this — the only scope in this API that fires for a repo
|
||||
// in an org created after this hook was registered, with no
|
||||
// per-org registration to keep in sync as orgs come and go.
|
||||
DeliveryKind::VcsActivity => ("push", HookScope::Instance),
|
||||
// Instance-wide too: any repo whose admin gave the
|
||||
// classifier's account access can request its review.
|
||||
DeliveryKind::ReviewRequest => ("pull_request_review_request", HookScope::Instance),
|
||||
};
|
||||
self.ensure_hook(&scope, &target_url, event, secret)
|
||||
let (events, scope) = hook_registration(kind);
|
||||
self.ensure_hook(&scope, &target_url, events, secret)
|
||||
.await
|
||||
.with_context(|| format!("register swarm webhook {target_url}"))?;
|
||||
}
|
||||
|
|
@ -1650,7 +1629,7 @@ impl Client {
|
|||
&self,
|
||||
scope: &HookScope<'_>,
|
||||
target_url: &str,
|
||||
event: &str,
|
||||
events: &[&str],
|
||||
secret: &str,
|
||||
) -> Result<()> {
|
||||
// ⚠️ Both non-matching arms log at a level the journal keeps, because
|
||||
|
|
@ -1667,15 +1646,27 @@ impl Client {
|
|||
// [`HookScope::extra_create_config`], which is where the instance
|
||||
// scope's own version of that went wrong), a non-zero count with no
|
||||
// match means the recorded url is not the one compared.
|
||||
match scope.list_hook_urls(&self.api).await {
|
||||
Ok(urls) => {
|
||||
if urls.iter().any(|u| u == target_url) {
|
||||
//
|
||||
// A listed hook missing one of `events` has its event list replaced,
|
||||
// so a hook registered before an event was added here gets it.
|
||||
match scope.list_hooks(&self.api).await {
|
||||
Ok(hooks) => {
|
||||
if let Some(hook) = hooks.iter().find(|h| h.url == target_url) {
|
||||
let missing = missing_events(events, &hook.events);
|
||||
if missing.is_empty() {
|
||||
tracing::debug!(%target_url, "swarm forge: webhook already registered");
|
||||
return Ok(());
|
||||
}
|
||||
let id = hook.id.with_context(|| {
|
||||
format!("the forge listed the hook for {target_url} without an id")
|
||||
})?;
|
||||
scope.edit_hook_events(&self.api, id, events).await?;
|
||||
tracing::info!(%target_url, ?missing, "swarm forge: webhook events updated");
|
||||
return Ok(());
|
||||
}
|
||||
tracing::info!(
|
||||
%target_url,
|
||||
listed = urls.len(),
|
||||
listed = hooks.len(),
|
||||
"swarm forge: no listed hook matches; registering"
|
||||
);
|
||||
}
|
||||
|
|
@ -1704,12 +1695,12 @@ impl Client {
|
|||
.with_context(|| format!("parse webhook target url {target_url}"))?,
|
||||
additional,
|
||||
},
|
||||
events: Some(vec![event.to_owned()]),
|
||||
events: Some(events.iter().map(|e| (*e).to_owned()).collect()),
|
||||
r#type: CreateHookOptionType::Forgejo,
|
||||
};
|
||||
match scope.create_hook(&self.api, hook).await {
|
||||
Ok(()) => {
|
||||
tracing::info!(%target_url, %event, "swarm forge: webhook registered");
|
||||
tracing::info!(%target_url, ?events, "swarm forge: webhook registered");
|
||||
Ok(())
|
||||
}
|
||||
Err(e) if is_already_exists(&e) => {
|
||||
|
|
@ -1744,6 +1735,60 @@ fn transitive_reach(start: i64, adj: &HashMap<i64, Vec<i64>>) -> usize {
|
|||
visited.len() - 1
|
||||
}
|
||||
|
||||
/// The events a swarm hook of `kind` is registered for, and the scope it
|
||||
/// lives on.
|
||||
fn hook_registration(kind: DeliveryKind) -> (&'static [&'static str], HookScope<'static>) {
|
||||
match kind {
|
||||
DeliveryKind::Knowledge => (
|
||||
&["push"],
|
||||
HookScope::Repo {
|
||||
org: KNOWLEDGE_ORG,
|
||||
repo: KNOWLEDGE_REPO,
|
||||
},
|
||||
),
|
||||
DeliveryKind::ConfigPr => (&["pull_request"], HookScope::Org { org: CONFIG_ORG }),
|
||||
// Instance-wide: commit/push activity is not scoped to one
|
||||
// org (unlike the two above) — it needs observing across
|
||||
// every repo the forge hosts, per the "no metric exists for
|
||||
// commits or pushes" tracker issue. Forgejo's
|
||||
// "global (system) webhook" (`admin_create_hook`) is exactly
|
||||
// this — the only scope in this API that fires for a repo
|
||||
// in an org created after this hook was registered, with no
|
||||
// per-org registration to keep in sync as orgs come and go.
|
||||
DeliveryKind::VcsActivity => (&["push"], HookScope::Instance),
|
||||
// Instance-wide too: any repo whose admin gave the
|
||||
// classifier's account access can request its review.
|
||||
DeliveryKind::ReviewRequest => (&["pull_request_review_request"], HookScope::Instance),
|
||||
// Instance-wide, for the notification router, which reads the assign
|
||||
// events. They are named: Forgejo's web form shows assignment as a
|
||||
// box of its own, so a hook edited there can drop it while keeping
|
||||
// `issues` or `pull_request`.
|
||||
DeliveryKind::Issues => (&["issues", "issue_assign"], HookScope::Instance),
|
||||
DeliveryKind::PullRequest => (
|
||||
&["pull_request", "pull_request_assign"],
|
||||
HookScope::Instance,
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
/// The events of `wanted` that a hook listed with `listed` lacks. Forgejo
|
||||
/// lists a hook by its specific event types, so `issue_assign` shows in the
|
||||
/// list on its own.
|
||||
fn missing_events<'a>(wanted: &[&'a str], listed: &[String]) -> Vec<&'a str> {
|
||||
wanted
|
||||
.iter()
|
||||
.filter(|w| !listed.iter().any(|l| l == *w))
|
||||
.copied()
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// One hook as [`HookScope::list_hooks`] reads it.
|
||||
struct ListedHook {
|
||||
id: Option<i64>,
|
||||
url: String,
|
||||
events: Vec<String>,
|
||||
}
|
||||
|
||||
/// Where a hook lives. The knowledge hook is repo-scoped and the config-PR
|
||||
/// hook is org-scoped — a hook on the wrong scope would never fire, and forgejo would
|
||||
/// report that as a perfectly healthy hook with no deliveries.
|
||||
|
|
@ -1775,7 +1820,7 @@ impl HookScope<'_> {
|
|||
/// hook — while `GET /admin/hooks` returns only webhooks with the flag
|
||||
/// set. Omitting it therefore breaks both halves at once: the hook is
|
||||
/// not the instance-wide one this scope exists for, and
|
||||
/// [`Self::list_hook_urls`] can never see it, so every process start
|
||||
/// [`Self::list_hooks`] can never see it, so every process start
|
||||
/// creates another one — twelve in a day, `listed=0` each time,
|
||||
/// before this key was sent.
|
||||
fn extra_create_config(&self) -> &'static [(&'static str, &'static str)] {
|
||||
|
|
@ -1785,8 +1830,8 @@ impl HookScope<'_> {
|
|||
}
|
||||
}
|
||||
|
||||
/// The `url` config value of every hook currently on this scope.
|
||||
async fn list_hook_urls(&self, api: &api::Api) -> Result<Vec<String>, ForgejoError> {
|
||||
/// Every hook currently on this scope that has a `url` config value.
|
||||
async fn list_hooks(&self, api: &api::Api) -> Result<Vec<ListedHook>, ForgejoError> {
|
||||
let hooks = match self {
|
||||
Self::Repo { org, repo } => {
|
||||
api.call(|api| Box::pin(api.repo_list_hooks(org, repo).all()))
|
||||
|
|
@ -1799,11 +1844,51 @@ impl HookScope<'_> {
|
|||
Self::Instance => api.call(|api| api.admin_list_hooks().into_future()).await?,
|
||||
};
|
||||
Ok(hooks
|
||||
.iter()
|
||||
.filter_map(|h| h.config.as_ref()?.get("url").cloned())
|
||||
.into_iter()
|
||||
.filter_map(|h| {
|
||||
Some(ListedHook {
|
||||
url: h.config.as_ref()?.get("url").cloned()?,
|
||||
id: h.id,
|
||||
events: h.events.unwrap_or_default(),
|
||||
})
|
||||
})
|
||||
.collect())
|
||||
}
|
||||
|
||||
/// Replace the event list of the hook `id` on this scope with `events`,
|
||||
/// leaving its url, secret and content type as they are.
|
||||
async fn edit_hook_events(
|
||||
&self,
|
||||
api: &api::Api,
|
||||
id: i64,
|
||||
events: &[&str],
|
||||
) -> Result<(), ForgejoError> {
|
||||
let edit = EditHookOption {
|
||||
active: None,
|
||||
authorization_header: None,
|
||||
branch_filter: None,
|
||||
config: None,
|
||||
events: Some(events.iter().map(|e| (*e).to_owned()).collect()),
|
||||
};
|
||||
match self {
|
||||
Self::Repo { org, repo } => api
|
||||
.call(|api| {
|
||||
api.repo_edit_hook(org, repo, id, edit.clone())
|
||||
.into_future()
|
||||
})
|
||||
.await
|
||||
.map(drop),
|
||||
Self::Org { org } => api
|
||||
.call(|api| api.org_edit_hook(org, id, edit.clone()).into_future())
|
||||
.await
|
||||
.map(drop),
|
||||
Self::Instance => api
|
||||
.call(|api| api.admin_edit_hook(id, edit.clone()).into_future())
|
||||
.await
|
||||
.map(drop),
|
||||
}
|
||||
}
|
||||
|
||||
async fn create_hook(
|
||||
&self,
|
||||
api: &api::Api,
|
||||
|
|
@ -2011,6 +2096,50 @@ impl Client {
|
|||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// The event list each swarm hook is registered with, as Forgejo names
|
||||
/// the event types (`modules/webhook/type.go`).
|
||||
#[test]
|
||||
fn each_swarm_hook_is_registered_for_its_named_events() {
|
||||
let events = |kind| hook_registration(kind).0.to_vec();
|
||||
assert_eq!(events(DeliveryKind::Knowledge), ["push"]);
|
||||
assert_eq!(events(DeliveryKind::ConfigPr), ["pull_request"]);
|
||||
assert_eq!(events(DeliveryKind::VcsActivity), ["push"]);
|
||||
assert_eq!(
|
||||
events(DeliveryKind::ReviewRequest),
|
||||
["pull_request_review_request"]
|
||||
);
|
||||
assert_eq!(events(DeliveryKind::Issues), ["issues", "issue_assign"]);
|
||||
assert_eq!(
|
||||
events(DeliveryKind::PullRequest),
|
||||
["pull_request", "pull_request_assign"]
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_listed_hook_lacking_a_wanted_event_is_updated_and_one_with_more_is_not() {
|
||||
let listed = |l: &[&str]| l.iter().map(|s| (*s).to_owned()).collect::<Vec<_>>();
|
||||
let wanted = ["issues", "issue_assign"];
|
||||
assert_eq!(
|
||||
missing_events(&wanted, &listed(&["issues"])),
|
||||
["issue_assign"]
|
||||
);
|
||||
assert_eq!(missing_events(&wanted, &listed(&[])), wanted);
|
||||
// How Forgejo lists a hook registered as `issues`.
|
||||
assert!(
|
||||
missing_events(
|
||||
&wanted,
|
||||
&listed(&[
|
||||
"issues",
|
||||
"issue_assign",
|
||||
"issue_label",
|
||||
"issue_milestone",
|
||||
"issue_comment"
|
||||
])
|
||||
)
|
||||
.is_empty()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn open_pulls_are_listed_newest_first_and_skip_one_with_no_number() {
|
||||
// The generated type insists on `requested_reviewers` being present.
|
||||
|
|
|
|||
120
swarm-controller/src/forge_events.rs
Normal file
120
swarm-controller/src/forge_events.rs
Normal file
|
|
@ -0,0 +1,120 @@
|
|||
//! Keeping verified `issues` and `pull_request` deliveries for the
|
||||
//! notification router.
|
||||
//!
|
||||
//! [`crate::webhook`] checks the HMAC, then [`publish`] puts the body,
|
||||
//! unchanged, on `swarm_wire::forge_events::subject(<event type>)` in the
|
||||
//! `SWARM_EVENTS` stream. The router reads that stream, so it needs no
|
||||
//! webhook secret and registers no hook of its own.
|
||||
//!
|
||||
//! The event type is Forgejo's `X-Forgejo-Event-Type` header (`issue_assign`,
|
||||
//! `pull_request_sync`, …), not `X-Forgejo-Event`, which folds those into
|
||||
//! `issues` and `pull_request`. It reaches the subject only once it is one
|
||||
//! plain token.
|
||||
|
||||
use axum::http::HeaderMap;
|
||||
use std::time::Duration;
|
||||
|
||||
/// The header naming the delivery's event type.
|
||||
const EVENT_TYPE_HEADER: &str = "x-forgejo-event-type";
|
||||
|
||||
/// The header naming the delivery. A redelivery carries the same one, and the
|
||||
/// stream drops a second message with the same id.
|
||||
const DELIVERY_HEADER: &str = "x-forgejo-delivery";
|
||||
|
||||
/// How long the stream may take to ack a publish. Inside Forgejo's own
|
||||
/// delivery timeout.
|
||||
const PUBLISH_DEADLINE: Duration = Duration::from_secs(5);
|
||||
|
||||
/// Why a delivery was not kept.
|
||||
#[derive(Debug, PartialEq, Eq)]
|
||||
pub(crate) enum NotKept {
|
||||
/// No `X-Forgejo-Event-Type`, or one that is not a single subject token.
|
||||
BadEventType,
|
||||
/// The controller has no queue, or the stream refused or timed out.
|
||||
Queue(String),
|
||||
}
|
||||
|
||||
/// The subject a delivery with `headers` goes to.
|
||||
fn subject(headers: &HeaderMap) -> Result<String, NotKept> {
|
||||
let event_type = headers
|
||||
.get(EVENT_TYPE_HEADER)
|
||||
.and_then(|v| v.to_str().ok())
|
||||
.filter(|t| swarm_wire::forge_events::is_subject_token(t))
|
||||
.ok_or(NotKept::BadEventType)?;
|
||||
Ok(swarm_wire::forge_events::subject(event_type))
|
||||
}
|
||||
|
||||
/// Keep one verified delivery in the stream, and wait for the stream's ack.
|
||||
pub(crate) async fn publish(
|
||||
state: &crate::routes::AppState,
|
||||
headers: &HeaderMap,
|
||||
body: axum::body::Bytes,
|
||||
) -> Result<(), NotKept> {
|
||||
let subject = subject(headers)?;
|
||||
let Some(status) = state.status.as_ref() else {
|
||||
return Err(NotKept::Queue("no swarm queue is configured".to_owned()));
|
||||
};
|
||||
let client = status.queue_client();
|
||||
swarm_queue_client::ensure_connected(&client).map_err(|e| NotKept::Queue(e.to_string()))?;
|
||||
let mut message = async_nats::jetstream::message::PublishMessage::build().payload(body);
|
||||
if let Some(id) = headers.get(DELIVERY_HEADER).and_then(|v| v.to_str().ok()) {
|
||||
message = message.message_id(id);
|
||||
}
|
||||
let kept = async {
|
||||
swarm_queue_client::forge_events::open_or_create(&client)
|
||||
.await
|
||||
.map_err(|e| swarm_queue_client::chain(&e))?;
|
||||
async_nats::jetstream::new(client.clone())
|
||||
.send_publish(subject.clone(), message)
|
||||
.await
|
||||
.map_err(|e| swarm_queue_client::chain(&e))?
|
||||
.await
|
||||
.map_err(|e| swarm_queue_client::chain(&e))
|
||||
};
|
||||
match tokio::time::timeout(PUBLISH_DEADLINE, kept).await {
|
||||
Ok(Ok(ack)) => {
|
||||
tracing::info!(%subject, seq = ack.sequence, duplicate = ack.duplicate, "webhook: forge event kept");
|
||||
Ok(())
|
||||
}
|
||||
Ok(Err(e)) => Err(NotKept::Queue(e)),
|
||||
Err(_) => Err(NotKept::Queue(format!(
|
||||
"the stream did not ack within {}s",
|
||||
PUBLISH_DEADLINE.as_secs()
|
||||
))),
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::{NotKept, subject};
|
||||
use axum::http::{HeaderMap, HeaderValue};
|
||||
|
||||
fn with_type(value: &str) -> HeaderMap {
|
||||
let mut headers = HeaderMap::new();
|
||||
headers.insert(
|
||||
"X-Forgejo-Event-Type",
|
||||
HeaderValue::from_str(value).expect("a header value"),
|
||||
);
|
||||
headers
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_event_type_header_names_the_subject() {
|
||||
assert_eq!(
|
||||
subject(&with_type("issue_assign")),
|
||||
Ok("$SWARM.events.forge.issue_assign".to_owned())
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_missing_or_multi_token_event_type_is_not_kept() {
|
||||
assert_eq!(subject(&HeaderMap::new()), Err(NotKept::BadEventType));
|
||||
for bad in ["issues.x", ">", "*", ""] {
|
||||
assert_eq!(
|
||||
subject(&with_type(bad)),
|
||||
Err(NotKept::BadEventType),
|
||||
"{bad:?}"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -42,6 +42,7 @@ mod claude_login;
|
|||
mod config_pr;
|
||||
mod forge;
|
||||
mod forge_account;
|
||||
mod forge_events;
|
||||
mod github_account;
|
||||
mod inference;
|
||||
mod issue_report;
|
||||
|
|
|
|||
|
|
@ -10,6 +10,10 @@
|
|||
//! the payload and emits a semantic message — *the knowledge repo changed*,
|
||||
//! *deploy agent X at rev Y* — addressed to the hives that need it. One place
|
||||
//! decides what a forge payload means, so no hive re-derives it.
|
||||
//!
|
||||
//! The `issues` and `pull_request` hooks are the exception: their bodies are
|
||||
//! kept unchanged in the forge events stream (`crate::forge_events`), and the
|
||||
//! notification router decides what they mean.
|
||||
|
||||
use anyhow::{Context as _, Result};
|
||||
use axum::{
|
||||
|
|
@ -225,6 +229,14 @@ pub(super) enum DeliveryKind {
|
|||
/// for the review classifier's account (`crate::review_classifier::review`).
|
||||
#[strum(serialize = "review-request")]
|
||||
ReviewRequest,
|
||||
/// Every `issues` event on every repo in the instance (opened, closed,
|
||||
/// assigned, …), kept for the notification router (`crate::forge_events`).
|
||||
#[strum(serialize = "issues")]
|
||||
Issues,
|
||||
/// Every `pull_request` event on every repo in the instance, kept the
|
||||
/// same way.
|
||||
#[strum(serialize = "pull-request")]
|
||||
PullRequest,
|
||||
}
|
||||
|
||||
/// The route prefix a registered `target_url` must point at.
|
||||
|
|
@ -240,11 +252,13 @@ impl DeliveryKind {
|
|||
/// Every kind, for the registration sweep. An array rather than a
|
||||
/// hand-written list at the call site, so adding a kind cannot leave one
|
||||
/// hook unregistered.
|
||||
pub(super) const ALL: [Self; 4] = [
|
||||
pub(super) const ALL: [Self; 6] = [
|
||||
Self::Knowledge,
|
||||
Self::ConfigPr,
|
||||
Self::VcsActivity,
|
||||
Self::ReviewRequest,
|
||||
Self::Issues,
|
||||
Self::PullRequest,
|
||||
];
|
||||
|
||||
/// The `target_url` to register with Forgejo for this kind, given the
|
||||
|
|
@ -352,18 +366,22 @@ pub(super) fn verify(
|
|||
/// comment for why both this and a periodic poll write the same cache), and a
|
||||
/// merge queues a deploy of the merged commit on the agent's hive. A
|
||||
/// `ReviewRequest` delivery naming the review classifier's account starts a
|
||||
/// review run in the background.
|
||||
/// review run in the background. An `Issues` or `PullRequest` delivery is
|
||||
/// kept in the forge events stream for the notification router
|
||||
/// (`crate::forge_events`).
|
||||
///
|
||||
/// Returns 200 on an accepted delivery so Forgejo does not retry. A refused
|
||||
/// one answers 401 (bad signature) or 503 (this daemon has no secret), and
|
||||
/// an unknown `{kind}` answers 404 rather than a cheerful 200.
|
||||
/// an unknown `{kind}` answers 404 rather than a cheerful 200. An `Issues` or
|
||||
/// `PullRequest` delivery the stream did not keep answers 503, so the forge's
|
||||
/// delivery log shows it failed and a redelivery can keep it.
|
||||
#[utoipa::path(
|
||||
post,
|
||||
path = "/webhook/forge/{kind}",
|
||||
params(
|
||||
("kind" = String, Path, description = "which swarm-wide hook this delivery arrived on: \
|
||||
`knowledge`, `config-pr`, `vcs-activity` or \
|
||||
`review-request`")
|
||||
`knowledge`, `config-pr`, `vcs-activity`, \
|
||||
`review-request`, `issues` or `pull-request`")
|
||||
),
|
||||
request_body(
|
||||
content = String,
|
||||
|
|
@ -376,7 +394,9 @@ pub(super) fn verify(
|
|||
(status = 200, description = "delivery accepted", body = String),
|
||||
(status = 401, description = "bad or missing HMAC signature"),
|
||||
(status = 404, description = "unknown hook kind in the path"),
|
||||
(status = 503, description = "HMAC secret unavailable at startup"),
|
||||
(status = 503, description = "HMAC secret unavailable at startup, or an `issues` or \
|
||||
`pull-request` delivery the forge events stream did not \
|
||||
keep"),
|
||||
),
|
||||
tag = "webhook"
|
||||
)]
|
||||
|
|
@ -430,6 +450,12 @@ pub(super) async fn post_webhook_forge(
|
|||
}
|
||||
}
|
||||
DeliveryKind::ReviewRequest => crate::review_classifier::review::on_delivery(&state, &body),
|
||||
DeliveryKind::Issues | DeliveryKind::PullRequest => {
|
||||
if let Err(not_kept) = crate::forge_events::publish(&state, &headers, body).await {
|
||||
tracing::warn!(kind = <&str>::from(kind), reason = ?not_kept, "webhook: forge event not kept");
|
||||
return (StatusCode::SERVICE_UNAVAILABLE, "forge event not kept").into_response();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
(StatusCode::OK, "ok").into_response()
|
||||
|
|
|
|||
|
|
@ -181,6 +181,11 @@ struct Args {
|
|||
#[arg(long = "claude-login-client")]
|
||||
claude_login_clients: Vec<String>,
|
||||
|
||||
/// Client ids that route forge events to agents' todo queues.
|
||||
/// Repeatable, empty by default, which denies such a client.
|
||||
#[arg(long = "notify-router-client")]
|
||||
notify_router_clients: Vec<String>,
|
||||
|
||||
/// Role on the secret store's `cert` auth mount this responder logs in
|
||||
/// as to read the callout keys, its own OIDC client secret and agent
|
||||
/// credentials. The store is found
|
||||
|
|
@ -247,7 +252,8 @@ async fn serve(args: Args) -> anyhow::Result<()> {
|
|||
.with_agent_token_subjects(args.agent_token_publish_subjects.clone())?
|
||||
.with_review_classifiers(args.review_classifier_clients.clone())
|
||||
.with_builders(args.builder_clients.clone())
|
||||
.with_claude_logins(args.claude_login_clients.clone());
|
||||
.with_claude_logins(args.claude_login_clients.clone())
|
||||
.with_notify_routers(args.notify_router_clients.clone());
|
||||
let http = reqwest::Client::new();
|
||||
let settings = swarm_secret_client::client::Settings::from_env()
|
||||
.context("reading the secret store's address and this responder's identity")?;
|
||||
|
|
|
|||
|
|
@ -52,6 +52,7 @@ pub struct Policy {
|
|||
review_classifiers: Vec<String>,
|
||||
builders: Vec<String>,
|
||||
claude_logins: Vec<String>,
|
||||
notify_routers: Vec<String>,
|
||||
}
|
||||
|
||||
/// Placeholder replaced with the hive's own name in `extra_hive_subjects`.
|
||||
|
|
@ -124,6 +125,7 @@ impl Policy {
|
|||
review_classifiers: Vec::new(),
|
||||
builders: Vec::new(),
|
||||
claude_logins: Vec::new(),
|
||||
notify_routers: Vec::new(),
|
||||
})
|
||||
}
|
||||
|
||||
|
|
@ -166,6 +168,13 @@ impl Policy {
|
|||
self
|
||||
}
|
||||
|
||||
/// Set the client ids that route forge events to agents' todo queues.
|
||||
#[must_use]
|
||||
pub fn with_notify_routers(mut self, clients: Vec<String>) -> Self {
|
||||
self.notify_routers = clients;
|
||||
self
|
||||
}
|
||||
|
||||
/// The permissions for an agent whose own credential was verified, or
|
||||
/// `None` when none are configured.
|
||||
///
|
||||
|
|
@ -253,6 +262,9 @@ impl Policy {
|
|||
if self.claude_logins.iter().any(|c| c == client_id) {
|
||||
return Some(Self::claude_login_permissions(client_id));
|
||||
}
|
||||
if self.notify_routers.iter().any(|c| c == client_id) {
|
||||
return Some(Self::notify_router_permissions(client_id));
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
|
|
@ -303,6 +315,56 @@ impl Policy {
|
|||
}
|
||||
}
|
||||
|
||||
/// What the notification router may do: drain its own consumer on the
|
||||
/// forge events stream, queue todo ops for any agent, keep its own state
|
||||
/// bucket, and list which agents exist.
|
||||
///
|
||||
/// - **Forge events.** Open or create the stream, create its one durable
|
||||
/// in the extended form whose last tokens are the filter (the server
|
||||
/// checks them against the payload), pull from it and ack. No bare
|
||||
/// `CONSUMER.CREATE`, no `CONSUMER.INFO`, no publish on the events: the
|
||||
/// controller is their one publisher.
|
||||
/// - **Todo queue.** Open or create `ROUTER_TODOS` and publish on every
|
||||
/// agent's subject: one per agent, so a wildcard, for the no-roster
|
||||
/// reason the reader's input grant gives. No other grant names these
|
||||
/// subjects, which is what makes the router their sole writer.
|
||||
/// - **Its state bucket.** Create, look up, read and write
|
||||
/// `notify-router-state`, and no other bucket.
|
||||
/// - **The roster.** `STREAM.INFO` on the agent-status bucket, which lists
|
||||
/// its keys (`<hive>.<agent>`) and writes none.
|
||||
fn notify_router_permissions(client_id: &str) -> Permissions {
|
||||
use swarm_wire::{forge_events, notify_router, router_todo};
|
||||
|
||||
let events = forge_events::STREAM;
|
||||
let consumer = forge_events::ROUTER_CONSUMER;
|
||||
let state = format!("KV_{}", notify_router::STATE_BUCKET);
|
||||
let mut publish = Self::jetstream_minimum().to_vec();
|
||||
publish.extend([
|
||||
format!("$JS.API.STREAM.INFO.{events}"),
|
||||
format!("$JS.API.STREAM.CREATE.{events}"),
|
||||
format!(
|
||||
"$JS.API.CONSUMER.CREATE.{events}.{consumer}.{}",
|
||||
forge_events::subject_wildcard()
|
||||
),
|
||||
format!("$JS.API.CONSUMER.MSG.NEXT.{events}.{consumer}"),
|
||||
format!("$JS.ACK.{events}.{consumer}.>"),
|
||||
format!("$JS.API.STREAM.INFO.{}", router_todo::STREAM),
|
||||
format!("$JS.API.STREAM.CREATE.{}", router_todo::STREAM),
|
||||
router_todo::subject_wildcard(),
|
||||
format!("$JS.API.STREAM.INFO.{state}"),
|
||||
format!("$JS.API.STREAM.CREATE.{state}"),
|
||||
format!("$JS.API.DIRECT.GET.{state}.>"),
|
||||
format!("$KV.{}.>", notify_router::STATE_BUCKET),
|
||||
format!("$JS.API.STREAM.INFO.{}", Self::agent_status_stream()),
|
||||
Self::inbox(client_id),
|
||||
]);
|
||||
Permissions {
|
||||
publish,
|
||||
subscribe: Some(vec![Self::inbox(client_id)]),
|
||||
responses: None,
|
||||
}
|
||||
}
|
||||
|
||||
/// What a hive may subscribe to: the two events the controller sends it,
|
||||
/// and its own reply inbox.
|
||||
///
|
||||
|
|
@ -666,6 +728,10 @@ impl Policy {
|
|||
swarm_wire::builder::ARCHETYPES_SUBJECT.to_owned(),
|
||||
// An operator's claude login step, sent as a request the same way.
|
||||
swarm_wire::claude_login::SUBJECT.to_owned(),
|
||||
// Verified forge deliveries, kept in the forge events stream for
|
||||
// the notification router. This client is their one publisher: the
|
||||
// router reads them and no hive or agent grant names them.
|
||||
swarm_wire::forge_events::subject_wildcard(),
|
||||
// The wanted-state buckets — one per hive, all created and written
|
||||
// by this single client.
|
||||
//
|
||||
|
|
@ -1642,6 +1708,159 @@ mod tests {
|
|||
}
|
||||
}
|
||||
|
||||
const NOTIFY_ROUTER: &str = "swarm-notify-router";
|
||||
|
||||
/// The three router-todo templates as `swarm-nats.nix` spells them.
|
||||
const ROUTER_TODO_TEMPLATES: [&str; 3] = [
|
||||
"$JS.API.CONSUMER.CREATE.ROUTER_TODOS.todo-{agent}.$SWARM.router.todo.{agent}",
|
||||
"$JS.API.CONSUMER.MSG.NEXT.ROUTER_TODOS.todo-{agent}",
|
||||
"$JS.ACK.ROUTER_TODOS.todo-{agent}.>",
|
||||
];
|
||||
|
||||
fn with_router() -> Policy {
|
||||
with_classifier()
|
||||
.with_builders(vec![BUILDER.to_owned()])
|
||||
.with_claude_logins(vec![CLAUDE_LOGIN.to_owned()])
|
||||
.with_notify_routers(vec![NOTIFY_ROUTER.to_owned()])
|
||||
.with_agent_token_subjects(ROUTER_TODO_TEMPLATES.map(str::to_owned).to_vec())
|
||||
.expect("per-agent templates are valid")
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_notify_router_drains_its_own_consumer_and_writes_only_its_own_state() {
|
||||
use swarm_queue_client::client_inbox_prefix;
|
||||
use swarm_wire::{forge_events, notify_router, router_todo};
|
||||
|
||||
let p = with_router()
|
||||
.permissions(NOTIFY_ROUTER)
|
||||
.expect("a configured router is admitted");
|
||||
let may = |subject: &str| p.publish.iter().any(|g| covers(g, subject));
|
||||
let events = forge_events::STREAM;
|
||||
let state = notify_router::STATE_BUCKET;
|
||||
for allowed in [
|
||||
format!(
|
||||
"$JS.API.CONSUMER.CREATE.{events}.notify-router.{}",
|
||||
forge_events::subject_wildcard()
|
||||
),
|
||||
format!("$JS.API.CONSUMER.MSG.NEXT.{events}.notify-router"),
|
||||
format!("$JS.ACK.{events}.notify-router.1.5.5.1700000000.0"),
|
||||
format!("$JS.API.STREAM.INFO.{events}"),
|
||||
format!("$JS.API.STREAM.CREATE.{}", router_todo::STREAM),
|
||||
router_todo::subject("atlas"),
|
||||
format!("$KV.{state}.42.5571"),
|
||||
format!("$JS.API.DIRECT.GET.KV_{state}.$KV.{state}.42.5571"),
|
||||
"$JS.API.STREAM.INFO.KV_agent-status".to_owned(),
|
||||
] {
|
||||
assert!(may(&allowed), "the router may not reach {allowed}");
|
||||
}
|
||||
for refused in [
|
||||
// The bare form, whose filter the server does not check.
|
||||
format!("$JS.API.CONSUMER.CREATE.{events}"),
|
||||
format!("$JS.API.CONSUMER.CREATE.{events}.other.$SWARM.events.forge.*"),
|
||||
format!("$JS.API.CONSUMER.DELETE.{events}.notify-router"),
|
||||
format!("$JS.API.STREAM.DELETE.{}", router_todo::STREAM),
|
||||
format!("$JS.API.STREAM.UPDATE.{}", router_todo::STREAM),
|
||||
format!(
|
||||
"$JS.API.CONSUMER.CREATE.{}.todo-atlas.{}",
|
||||
router_todo::STREAM,
|
||||
router_todo::subject("atlas")
|
||||
),
|
||||
forge_events::subject("issues"),
|
||||
"$KV.agent-status.alpha.atlas".to_owned(),
|
||||
"$KV.agent-schedules.atlas".to_owned(),
|
||||
swarm_queue_client::agent_input_subject("atlas"),
|
||||
"$JS.API.STREAM.CREATE.KV_agent-status".to_owned(),
|
||||
] {
|
||||
assert!(!may(&refused), "the router may reach {refused}");
|
||||
}
|
||||
let inbox = client_inbox_prefix(NOTIFY_ROUTER);
|
||||
assert_eq!(p.subscribe, Some(vec![format!("{inbox}.>")]));
|
||||
assert_eq!(p.responses, None);
|
||||
assert_eq!(policy().permissions(NOTIFY_ROUTER), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn only_the_router_may_queue_an_agents_todo_and_only_the_reader_may_keep_a_forge_event() {
|
||||
use swarm_wire::{forge_events, router_todo};
|
||||
|
||||
let p = with_router();
|
||||
let todo = router_todo::subject("atlas");
|
||||
let event = forge_events::subject("issue_assign");
|
||||
let grants = [
|
||||
(
|
||||
"the reader",
|
||||
p.permissions("swarm-controller").expect("reader"),
|
||||
),
|
||||
("a hive", p.permissions("hive-alpha").expect("a hive")),
|
||||
("the router", p.permissions(NOTIFY_ROUTER).expect("router")),
|
||||
("the minter", p.permissions(CLAUDE_LOGIN).expect("minter")),
|
||||
("the builder", p.permissions(BUILDER).expect("builder")),
|
||||
(
|
||||
"the classifier",
|
||||
p.permissions(CLASSIFIER).expect("classifier"),
|
||||
),
|
||||
("atlas", p.agent_token_permissions("atlas").expect("atlas")),
|
||||
];
|
||||
for (who, grant) in &grants {
|
||||
let queues = grant.publish.iter().any(|g| covers(g, &todo));
|
||||
let keeps = grant.publish.iter().any(|g| covers(g, &event));
|
||||
assert_eq!(queues, *who == "the router", "{who} queues: {queues}");
|
||||
assert_eq!(keeps, *who == "the reader", "{who} keeps events: {keeps}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_agent_may_drain_only_its_own_router_todo_consumer() {
|
||||
use swarm_wire::router_todo::{STREAM, consumer_name, subject};
|
||||
|
||||
let p = with_router();
|
||||
let atlas = p.agent_token_permissions("atlas").expect("configured");
|
||||
let own_create = format!(
|
||||
"$JS.API.CONSUMER.CREATE.{STREAM}.{}.{}",
|
||||
consumer_name("atlas"),
|
||||
subject("atlas")
|
||||
);
|
||||
assert_eq!(
|
||||
templated(&atlas),
|
||||
vec![
|
||||
own_create.clone(),
|
||||
"$JS.API.CONSUMER.MSG.NEXT.ROUTER_TODOS.todo-atlas".to_owned(),
|
||||
"$JS.ACK.ROUTER_TODOS.todo-atlas.>".to_owned(),
|
||||
]
|
||||
);
|
||||
// Control: argus's expansion names argus.
|
||||
let argus = p.agent_token_permissions("argus").expect("configured");
|
||||
assert!(
|
||||
argus
|
||||
.publish
|
||||
.contains(&"$JS.API.CONSUMER.MSG.NEXT.ROUTER_TODOS.todo-argus".to_owned())
|
||||
);
|
||||
let may = |s: &str| atlas.publish.iter().any(|g| covers(g, s));
|
||||
assert!(may(&own_create));
|
||||
assert!(may("$JS.ACK.ROUTER_TODOS.todo-atlas.1.3.3.1700000000.0"));
|
||||
for refused in [
|
||||
format!("$JS.API.CONSUMER.CREATE.{STREAM}"),
|
||||
format!("$JS.API.CONSUMER.CREATE.{STREAM}.todo-atlas"),
|
||||
format!(
|
||||
"$JS.API.CONSUMER.CREATE.{STREAM}.todo-atlas.{}",
|
||||
subject("argus")
|
||||
),
|
||||
format!("$JS.API.CONSUMER.CREATE.{STREAM}.todo-atlas.$SWARM.router.todo.*"),
|
||||
format!(
|
||||
"$JS.API.CONSUMER.CREATE.{STREAM}.todo-argus.{}",
|
||||
subject("argus")
|
||||
),
|
||||
format!("$JS.API.CONSUMER.MSG.NEXT.{STREAM}.todo-argus"),
|
||||
format!("$JS.API.CONSUMER.DELETE.{STREAM}.todo-atlas"),
|
||||
format!("$JS.API.CONSUMER.INFO.{STREAM}.todo-atlas"),
|
||||
format!("$JS.ACK.{STREAM}.todo-argus.1.3.3.1700000000.0"),
|
||||
format!("$JS.API.STREAM.INFO.{STREAM}"),
|
||||
subject("atlas"),
|
||||
] {
|
||||
assert!(!may(&refused), "atlas's grant covers {refused}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_prefix_alone_names_no_hive() {
|
||||
// `hive-` would otherwise produce `$KV.hive-status.`, a subject nobody
|
||||
|
|
|
|||
34
swarm-notify-router/Cargo.toml
Normal file
34
swarm-notify-router/Cargo.toml
Normal file
|
|
@ -0,0 +1,34 @@
|
|||
[package]
|
||||
name = "swarm-notify-router"
|
||||
version.workspace = true
|
||||
readme = "README.md"
|
||||
edition.workspace = true
|
||||
|
||||
[[bin]]
|
||||
name = "swarm-notify-router"
|
||||
path = "src/main.rs"
|
||||
|
||||
[dependencies]
|
||||
anyhow.workspace = true
|
||||
# `kv` for its state bucket; `jetstream` comes with it.
|
||||
async-nats = { workspace = true, features = ["kv"] }
|
||||
# StreamExt / TryStreamExt on the pulled batch and the subject listing.
|
||||
futures-util.workspace = true
|
||||
hive-log.workspace = true
|
||||
serde.workspace = true
|
||||
serde_json.workspace = true
|
||||
# The connect, and the two streams' configs it shares with the controller and
|
||||
# the agent harness.
|
||||
swarm-queue-client = { workspace = true, features = [
|
||||
"forge-events",
|
||||
"router-todo",
|
||||
] }
|
||||
# Its queue client secret, read with its own store leaf.
|
||||
swarm-secret-client.workspace = true
|
||||
# The stream, subject, bucket and op names.
|
||||
swarm-wire.workspace = true
|
||||
tokio.workspace = true
|
||||
tracing.workspace = true
|
||||
|
||||
[lints]
|
||||
workspace = true
|
||||
8
swarm-notify-router/README.md
Normal file
8
swarm-notify-router/README.md
Normal file
|
|
@ -0,0 +1,8 @@
|
|||
# swarm-notify-router
|
||||
|
||||
Routes forge events to agents' todo lists. It drains its consumer on the
|
||||
forge events stream that swarm-controller fills from its webhooks, runs its
|
||||
flows over each delivery, and queues the todo ops they emit on each agent's
|
||||
subject of the `ROUTER_TODOS` stream. Each agent's harness applies them.
|
||||
|
||||
Design: [`docs/swarm/notify-router.md`](../docs/swarm/notify-router.md).
|
||||
233
swarm-notify-router/src/event.rs
Normal file
233
swarm-notify-router/src/event.rs
Normal file
|
|
@ -0,0 +1,233 @@
|
|||
//! A forge delivery as the fields a flow reads, and the per-issue snapshot
|
||||
//! that fills in what the delivery leaves out.
|
||||
//!
|
||||
//! Forgejo's `issue_assign` and `pull_request_assign` deliveries carry the
|
||||
//! assignees *after* the change and not which one changed. [`Snapshot`] holds
|
||||
//! the assignees the router last saw on an assign delivery, so the added and
|
||||
//! removed ones are the difference. With no snapshot yet, every current
|
||||
//! assignee counts as added and none as removed: an unassign is noticed only
|
||||
//! on an issue the router has seen an assign delivery for.
|
||||
|
||||
use std::collections::BTreeSet;
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
use serde_json::Value as Json;
|
||||
|
||||
use crate::flow::{Fields, Value};
|
||||
|
||||
/// The parts of an `issues` or `pull_request` family delivery the router
|
||||
/// uses.
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct Delivery {
|
||||
pub action: String,
|
||||
pub repo: String,
|
||||
pub repo_id: i64,
|
||||
pub number: i64,
|
||||
pub title: String,
|
||||
pub url: String,
|
||||
pub sender: String,
|
||||
pub is_pull: bool,
|
||||
pub assignees: BTreeSet<String>,
|
||||
}
|
||||
|
||||
impl Delivery {
|
||||
/// Read a delivery body, or `None` when it names no issue or PR.
|
||||
pub fn parse(body: &[u8]) -> Option<Self> {
|
||||
let json: Json = serde_json::from_slice(body).ok()?;
|
||||
let (item, is_pull) = match json.get("pull_request") {
|
||||
Some(pr) if pr.is_object() => (pr, true),
|
||||
_ => (json.get("issue")?, false),
|
||||
};
|
||||
let text = |v: &Json, k: &str| {
|
||||
v.get(k)
|
||||
.and_then(Json::as_str)
|
||||
.unwrap_or_default()
|
||||
.to_owned()
|
||||
};
|
||||
let repository = json.get("repository")?;
|
||||
let assignees = item
|
||||
.get("assignees")
|
||||
.and_then(Json::as_array)
|
||||
.map(|a| {
|
||||
a.iter()
|
||||
.filter_map(|u| u.get("login").and_then(Json::as_str))
|
||||
.map(str::to_owned)
|
||||
.collect()
|
||||
})
|
||||
.unwrap_or_default();
|
||||
Some(Self {
|
||||
action: text(&json, "action"),
|
||||
repo: text(repository, "full_name"),
|
||||
repo_id: repository.get("id")?.as_i64()?,
|
||||
number: item.get("number")?.as_i64()?,
|
||||
title: text(item, "title"),
|
||||
url: text(item, "html_url"),
|
||||
sender: json
|
||||
.get("sender")
|
||||
.map(|s| text(s, "login"))
|
||||
.unwrap_or_default(),
|
||||
is_pull,
|
||||
assignees,
|
||||
})
|
||||
}
|
||||
|
||||
/// The state bucket key for this delivery's issue: `<repo id>.<number>`,
|
||||
/// numeric so it is always a valid key whatever the repo is named.
|
||||
pub fn state_key(&self) -> String {
|
||||
format!("{}.{}", self.repo_id, self.number)
|
||||
}
|
||||
}
|
||||
|
||||
/// What the router keeps per issue in its state bucket.
|
||||
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct Snapshot {
|
||||
/// The assignees after the last assign delivery seen, less those that
|
||||
/// were no agent of the roster then, or `None` before the first.
|
||||
#[serde(default)]
|
||||
pub assignees: Option<BTreeSet<String>>,
|
||||
/// The agents holding a router todo for this issue.
|
||||
#[serde(default)]
|
||||
pub holders: BTreeSet<String>,
|
||||
}
|
||||
|
||||
/// Whether `event_type` is an assign delivery, whose assignee list is the
|
||||
/// set after the change.
|
||||
pub fn is_assign(event_type: &str) -> bool {
|
||||
matches!(event_type, "issue_assign" | "pull_request_assign")
|
||||
}
|
||||
|
||||
/// The fields a flow reads for `delivery`, given the issue's `snapshot`.
|
||||
pub fn fields(event_type: &str, delivery: &Delivery, snapshot: &Snapshot) -> Fields {
|
||||
let (added, removed) = if is_assign(event_type) {
|
||||
match &snapshot.assignees {
|
||||
Some(before) => (
|
||||
delivery.assignees.difference(before).cloned().collect(),
|
||||
before.difference(&delivery.assignees).cloned().collect(),
|
||||
),
|
||||
None => (delivery.assignees.iter().cloned().collect(), Vec::new()),
|
||||
}
|
||||
} else {
|
||||
(Vec::new(), Vec::new())
|
||||
};
|
||||
let text = |s: &str| Value::Text(s.to_owned());
|
||||
Fields::from([
|
||||
("event", text(event_type)),
|
||||
("action", text(&delivery.action)),
|
||||
("kind", text(if delivery.is_pull { "PR" } else { "issue" })),
|
||||
("repo", text(&delivery.repo)),
|
||||
("number", Value::Text(delivery.number.to_string())),
|
||||
("title", text(&delivery.title)),
|
||||
("url", text(&delivery.url)),
|
||||
("sender", text(&delivery.sender)),
|
||||
(
|
||||
"assignees",
|
||||
Value::List(delivery.assignees.iter().cloned().collect()),
|
||||
),
|
||||
("assignees_added", Value::List(added)),
|
||||
("assignees_removed", Value::List(removed)),
|
||||
(
|
||||
"todo_holders",
|
||||
Value::List(snapshot.holders.iter().cloned().collect()),
|
||||
),
|
||||
])
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn issue_assign(action: &str, assignees: &[&str]) -> Vec<u8> {
|
||||
let users: Vec<Json> = assignees
|
||||
.iter()
|
||||
.map(|a| serde_json::json!({ "login": a }))
|
||||
.collect();
|
||||
serde_json::to_vec(&serde_json::json!({
|
||||
"action": action,
|
||||
"number": 7,
|
||||
"issue": {
|
||||
"number": 7,
|
||||
"title": "router",
|
||||
"html_url": "https://forge/issue-7",
|
||||
"assignees": users,
|
||||
},
|
||||
"repository": { "id": 42, "full_name": "hyperhive/hyperhive" },
|
||||
"sender": { "login": "mara" },
|
||||
}))
|
||||
.expect("encodes")
|
||||
}
|
||||
|
||||
fn list(fields: &Fields, name: &str) -> Vec<String> {
|
||||
match fields.get(name) {
|
||||
Some(Value::List(v)) => v.clone(),
|
||||
other => panic!("{name}: {other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_issue_delivery_reads_back_its_fields() {
|
||||
let d = Delivery::parse(&issue_assign("assigned", &["atlas", "iris"])).expect("parses");
|
||||
assert_eq!(d.repo, "hyperhive/hyperhive");
|
||||
assert_eq!(d.number, 7);
|
||||
assert!(!d.is_pull);
|
||||
assert_eq!(d.state_key(), "42.7");
|
||||
assert_eq!(
|
||||
d.assignees,
|
||||
BTreeSet::from(["atlas".to_owned(), "iris".to_owned()])
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_pull_request_delivery_reads_the_pull_request_object() {
|
||||
let body = serde_json::to_vec(&serde_json::json!({
|
||||
"action": "closed",
|
||||
"number": 7,
|
||||
"pull_request": { "number": 7, "title": "t", "html_url": "u", "assignees": null },
|
||||
"repository": { "id": 1, "full_name": "o/r" },
|
||||
"sender": { "login": "s" },
|
||||
}))
|
||||
.expect("encodes");
|
||||
let d = Delivery::parse(&body).expect("parses");
|
||||
assert!(d.is_pull);
|
||||
assert_eq!(d.number, 7);
|
||||
assert!(d.assignees.is_empty());
|
||||
assert_eq!(Delivery::parse(b"{\"ref\":\"main\"}"), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_snapshot_turns_the_after_list_into_added_and_removed() {
|
||||
let d = Delivery::parse(&issue_assign("unassigned", &["iris"])).expect("parses");
|
||||
let before = Snapshot {
|
||||
assignees: Some(BTreeSet::from(["atlas".to_owned(), "iris".to_owned()])),
|
||||
holders: BTreeSet::new(),
|
||||
};
|
||||
let f = fields("issue_assign", &d, &before);
|
||||
assert_eq!(list(&f, "assignees_removed"), ["atlas"]);
|
||||
assert!(list(&f, "assignees_added").is_empty());
|
||||
|
||||
let d = Delivery::parse(&issue_assign("assigned", &["atlas", "iris"])).expect("parses");
|
||||
let before = Snapshot {
|
||||
assignees: Some(BTreeSet::from(["iris".to_owned()])),
|
||||
holders: BTreeSet::new(),
|
||||
};
|
||||
assert_eq!(
|
||||
list(&fields("issue_assign", &d, &before), "assignees_added"),
|
||||
["atlas"]
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn with_no_snapshot_every_assignee_is_added_and_none_removed() {
|
||||
let d = Delivery::parse(&issue_assign("assigned", &["atlas"])).expect("parses");
|
||||
let f = fields("issue_assign", &d, &Snapshot::default());
|
||||
assert_eq!(list(&f, "assignees_added"), ["atlas"]);
|
||||
assert!(list(&f, "assignees_removed").is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_delivery_that_is_not_an_assign_adds_and_removes_nobody() {
|
||||
let d = Delivery::parse(&issue_assign("closed", &["atlas"])).expect("parses");
|
||||
let f = fields("issues", &d, &Snapshot::default());
|
||||
assert!(list(&f, "assignees_added").is_empty());
|
||||
assert!(list(&f, "assignees_removed").is_empty());
|
||||
}
|
||||
}
|
||||
518
swarm-notify-router/src/flow.rs
Normal file
518
swarm-notify-router/src/flow.rs
Normal file
|
|
@ -0,0 +1,518 @@
|
|||
//! Flows: what the router does with an event.
|
||||
//!
|
||||
//! A flow is a graph of nodes joined by edges, rooted at one `source` node.
|
||||
//! A `match` node passes when every one of its conditions holds, and then
|
||||
//! each of its children runs, so one event can reach several branches. A
|
||||
//! `sink` node is a leaf and emits one todo op per recipient.
|
||||
//!
|
||||
//! Flows are trees: one source, one parent for every other node, every node
|
||||
//! reachable from the source. [`Flow::validate`] checks that, and the router
|
||||
//! runs only flows that pass.
|
||||
//!
|
||||
//! The built-in flows ([`builtins`]) are ordinary flows defined in code.
|
||||
|
||||
use std::collections::{BTreeMap, BTreeSet};
|
||||
|
||||
use swarm_wire::router_todo::{Op, RouterTodoOp};
|
||||
|
||||
/// The most nodes one flow may have.
|
||||
const MAX_NODES: usize = 64;
|
||||
|
||||
/// The deepest a node may sit below the source.
|
||||
const MAX_DEPTH: usize = 16;
|
||||
|
||||
/// One event, as the fields a flow reads.
|
||||
pub type Fields = BTreeMap<&'static str, Value>;
|
||||
|
||||
/// A field's value.
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub enum Value {
|
||||
Text(String),
|
||||
List(Vec<String>),
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct Flow {
|
||||
pub id: String,
|
||||
pub nodes: Vec<Node>,
|
||||
pub edges: Vec<Edge>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct Node {
|
||||
pub id: String,
|
||||
pub kind: NodeKind,
|
||||
}
|
||||
|
||||
/// A directed edge, parent to child.
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct Edge {
|
||||
pub from: String,
|
||||
pub to: String,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
pub enum NodeKind {
|
||||
/// The root: the forge event types the flow starts from.
|
||||
Source { events: Vec<String> },
|
||||
/// Passes when every condition holds.
|
||||
Match { when: Vec<Condition> },
|
||||
/// Emits a todo op to each recipient.
|
||||
Todo(TodoSink),
|
||||
}
|
||||
|
||||
/// `field` equals `value`. A list field holds when one of its items does.
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct Condition {
|
||||
pub field: String,
|
||||
pub equals: String,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct TodoSink {
|
||||
/// The field naming the recipients, a list of agent names.
|
||||
pub to: String,
|
||||
pub op: Op,
|
||||
/// The todo key, with `{field}` placeholders.
|
||||
pub key: String,
|
||||
/// For a clear: `key` is a prefix.
|
||||
pub prefix: bool,
|
||||
/// The todo text, with `{field}` placeholders. Unused for a clear.
|
||||
pub text: String,
|
||||
}
|
||||
|
||||
/// One op the walk emitted, for one recipient.
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct Action {
|
||||
pub flow: String,
|
||||
pub node: String,
|
||||
pub agent: String,
|
||||
pub op: RouterTodoOp,
|
||||
}
|
||||
|
||||
impl Flow {
|
||||
/// Check the flow is a tree within the caps, or say why not.
|
||||
pub fn validate(&self) -> Result<(), String> {
|
||||
if self.nodes.len() > MAX_NODES {
|
||||
return Err(format!("{} nodes, more than {MAX_NODES}", self.nodes.len()));
|
||||
}
|
||||
let mut ids = BTreeSet::new();
|
||||
for node in &self.nodes {
|
||||
if !ids.insert(node.id.as_str()) {
|
||||
return Err(format!("node id {} is used twice", node.id));
|
||||
}
|
||||
}
|
||||
for edge in &self.edges {
|
||||
for end in [&edge.from, &edge.to] {
|
||||
if !ids.contains(end.as_str()) {
|
||||
return Err(format!("an edge names {end}, which is no node"));
|
||||
}
|
||||
}
|
||||
}
|
||||
let roots: Vec<&Node> = self
|
||||
.nodes
|
||||
.iter()
|
||||
.filter(|n| matches!(n.kind, NodeKind::Source { .. }))
|
||||
.collect();
|
||||
let [root] = roots.as_slice() else {
|
||||
return Err(format!("{} source nodes, not one", roots.len()));
|
||||
};
|
||||
for node in &self.nodes {
|
||||
if matches!(node.kind, NodeKind::Todo(_)) && !self.children(&node.id).is_empty() {
|
||||
return Err(format!("sink {} has children", node.id));
|
||||
}
|
||||
}
|
||||
// From the one root, every node reached exactly once: a node reached
|
||||
// twice has a second parent or sits on a cycle, and one not reached is
|
||||
// cut off.
|
||||
let mut reached = 0;
|
||||
let mut stack = vec![(root.id.as_str(), 0)];
|
||||
while let Some((id, depth)) = stack.pop() {
|
||||
if depth > MAX_DEPTH {
|
||||
return Err(format!("node {id} is deeper than {MAX_DEPTH}"));
|
||||
}
|
||||
reached += 1;
|
||||
if reached > self.nodes.len() {
|
||||
return Err(
|
||||
"a node is reached twice: it has two parents or sits on a cycle".to_owned(),
|
||||
);
|
||||
}
|
||||
stack.extend(self.children(id).into_iter().map(|c| (c, depth + 1)));
|
||||
}
|
||||
if reached != self.nodes.len() {
|
||||
return Err(format!(
|
||||
"{} of {} nodes are not reachable from the source",
|
||||
self.nodes.len() - reached,
|
||||
self.nodes.len()
|
||||
));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn children(&self, id: &str) -> Vec<&str> {
|
||||
self.edges
|
||||
.iter()
|
||||
.filter(|e| e.from == id)
|
||||
.map(|e| e.to.as_str())
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn node(&self, id: &str) -> Option<&Node> {
|
||||
self.nodes.iter().find(|n| n.id == id)
|
||||
}
|
||||
|
||||
/// Walk the flow for one event of type `event_type` and return the ops it
|
||||
/// emits. Assumes [`Self::validate`] passed.
|
||||
pub fn evaluate(&self, event_type: &str, fields: &Fields) -> Vec<Action> {
|
||||
let Some(root) = self
|
||||
.nodes
|
||||
.iter()
|
||||
.find(|n| matches!(&n.kind, NodeKind::Source { events } if events.iter().any(|e| e == event_type)))
|
||||
else {
|
||||
return Vec::new();
|
||||
};
|
||||
let mut actions = Vec::new();
|
||||
let mut stack = vec![root.id.as_str()];
|
||||
while let Some(id) = stack.pop() {
|
||||
let Some(node) = self.node(id) else { continue };
|
||||
match &node.kind {
|
||||
NodeKind::Source { .. } => stack.extend(self.children(id)),
|
||||
NodeKind::Match { when } => {
|
||||
if when.iter().all(|c| holds(c, fields)) {
|
||||
stack.extend(self.children(id));
|
||||
}
|
||||
}
|
||||
NodeKind::Todo(sink) => actions.extend(self.emit(node, sink, fields)),
|
||||
}
|
||||
}
|
||||
actions
|
||||
}
|
||||
|
||||
fn emit(&self, node: &Node, sink: &TodoSink, fields: &Fields) -> Vec<Action> {
|
||||
let recipients = match fields.get(sink.to.as_str()) {
|
||||
Some(Value::List(items)) => items.clone(),
|
||||
Some(Value::Text(one)) => vec![one.clone()],
|
||||
None => Vec::new(),
|
||||
};
|
||||
let op = RouterTodoOp {
|
||||
op: sink.op,
|
||||
key: expand(&sink.key, fields),
|
||||
prefix: sink.op == Op::Clear && sink.prefix,
|
||||
summary: match sink.op {
|
||||
Op::Upsert => expand(&sink.text, fields),
|
||||
Op::Clear => String::new(),
|
||||
},
|
||||
source: format!("router:{}", self.id),
|
||||
};
|
||||
recipients
|
||||
.into_iter()
|
||||
.map(|agent| Action {
|
||||
flow: self.id.clone(),
|
||||
node: node.id.clone(),
|
||||
agent,
|
||||
op: op.clone(),
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
}
|
||||
|
||||
fn holds(condition: &Condition, fields: &Fields) -> bool {
|
||||
match fields.get(condition.field.as_str()) {
|
||||
Some(Value::Text(v)) => *v == condition.equals,
|
||||
Some(Value::List(items)) => items.contains(&condition.equals),
|
||||
None => false,
|
||||
}
|
||||
}
|
||||
|
||||
/// `template` with each `{field}` replaced by that text field's value. A list
|
||||
/// field is joined with `, `; an unknown name is left as written.
|
||||
pub fn expand(template: &str, fields: &Fields) -> String {
|
||||
let mut out = String::with_capacity(template.len());
|
||||
let mut rest = template;
|
||||
while let Some(open) = rest.find('{') {
|
||||
out.push_str(&rest[..open]);
|
||||
let after = &rest[open + 1..];
|
||||
let Some(close) = after.find('}') else {
|
||||
out.push_str(&rest[open..]);
|
||||
return out;
|
||||
};
|
||||
let name = &after[..close];
|
||||
match fields.get(name) {
|
||||
Some(Value::Text(v)) => out.push_str(v),
|
||||
Some(Value::List(items)) => out.push_str(&items.join(", ")),
|
||||
None => {
|
||||
out.push('{');
|
||||
out.push_str(name);
|
||||
out.push('}');
|
||||
}
|
||||
}
|
||||
rest = &after[close + 1..];
|
||||
}
|
||||
out.push_str(rest);
|
||||
out
|
||||
}
|
||||
|
||||
/// The flows the router ships with.
|
||||
///
|
||||
/// - `assigned`: a newly assigned agent gets a todo for the issue or PR.
|
||||
/// - `unassigned`: an agent taken off it gets that todo rewritten to say so.
|
||||
/// Both write the same key, so the second replaces the first.
|
||||
/// - `closed`: closing it clears every router todo the router queued for it.
|
||||
pub fn builtins() -> Vec<Flow> {
|
||||
let assignment_key = "forge:{repo}#{number}:assignment";
|
||||
vec![
|
||||
chain(
|
||||
"assigned",
|
||||
&["issue_assign", "pull_request_assign"],
|
||||
"assigned",
|
||||
TodoSink {
|
||||
to: "assignees_added".to_owned(),
|
||||
op: Op::Upsert,
|
||||
key: assignment_key.to_owned(),
|
||||
prefix: false,
|
||||
text: "you are assigned to {kind} {repo}#{number}: {title} ({url})".to_owned(),
|
||||
},
|
||||
),
|
||||
chain(
|
||||
"unassigned",
|
||||
&["issue_assign", "pull_request_assign"],
|
||||
"unassigned",
|
||||
TodoSink {
|
||||
to: "assignees_removed".to_owned(),
|
||||
op: Op::Upsert,
|
||||
key: assignment_key.to_owned(),
|
||||
prefix: false,
|
||||
text: "you are not assigned to {kind} {repo}#{number} anymore: {title} ({url})"
|
||||
.to_owned(),
|
||||
},
|
||||
),
|
||||
chain(
|
||||
"closed",
|
||||
&["issues", "pull_request"],
|
||||
"closed",
|
||||
TodoSink {
|
||||
to: "todo_holders".to_owned(),
|
||||
op: Op::Clear,
|
||||
key: "forge:{repo}#{number}:".to_owned(),
|
||||
prefix: true,
|
||||
text: String::new(),
|
||||
},
|
||||
),
|
||||
]
|
||||
}
|
||||
|
||||
/// source(`events`) → match(`action` = `action`) → `sink`.
|
||||
fn chain(id: &str, events: &[&str], action: &str, sink: TodoSink) -> Flow {
|
||||
let node = |id: &str, kind| Node {
|
||||
id: id.to_owned(),
|
||||
kind,
|
||||
};
|
||||
let edge = |from: &str, to: &str| Edge {
|
||||
from: from.to_owned(),
|
||||
to: to.to_owned(),
|
||||
};
|
||||
Flow {
|
||||
id: id.to_owned(),
|
||||
nodes: vec![
|
||||
node(
|
||||
"source",
|
||||
NodeKind::Source {
|
||||
events: events.iter().map(|e| (*e).to_owned()).collect(),
|
||||
},
|
||||
),
|
||||
node(
|
||||
"match",
|
||||
NodeKind::Match {
|
||||
when: vec![Condition {
|
||||
field: "action".to_owned(),
|
||||
equals: action.to_owned(),
|
||||
}],
|
||||
},
|
||||
),
|
||||
node("sink", NodeKind::Todo(sink)),
|
||||
],
|
||||
edges: vec![edge("source", "match"), edge("match", "sink")],
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn text(v: &str) -> Value {
|
||||
Value::Text(v.to_owned())
|
||||
}
|
||||
|
||||
fn list(v: &[&str]) -> Value {
|
||||
Value::List(v.iter().map(|s| (*s).to_owned()).collect())
|
||||
}
|
||||
|
||||
fn event(action: &str, added: &[&str], removed: &[&str], holders: &[&str]) -> Fields {
|
||||
Fields::from([
|
||||
("action", text(action)),
|
||||
("kind", text("issue")),
|
||||
("repo", text("hyperhive/hyperhive")),
|
||||
("number", text("7")),
|
||||
("title", text("router")),
|
||||
("url", text("https://forge/7")),
|
||||
("assignees_added", list(added)),
|
||||
("assignees_removed", list(removed)),
|
||||
("todo_holders", list(holders)),
|
||||
])
|
||||
}
|
||||
|
||||
fn flow(id: &str) -> Flow {
|
||||
builtins()
|
||||
.into_iter()
|
||||
.find(|f| f.id == id)
|
||||
.expect("a built-in")
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn every_builtin_is_a_valid_tree() {
|
||||
for f in builtins() {
|
||||
assert_eq!(f.validate(), Ok(()), "{}", f.id);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn assigning_queues_a_todo_for_each_newly_assigned_agent_only() {
|
||||
let got = flow("assigned").evaluate(
|
||||
"issue_assign",
|
||||
&event("assigned", &["atlas", "iris"], &[], &[]),
|
||||
);
|
||||
let agents: Vec<&str> = got.iter().map(|a| a.agent.as_str()).collect();
|
||||
assert_eq!(agents, ["atlas", "iris"]);
|
||||
assert_eq!(got[0].op.op, Op::Upsert);
|
||||
assert_eq!(got[0].op.key, "forge:hyperhive/hyperhive#7:assignment");
|
||||
assert_eq!(
|
||||
got[0].op.summary,
|
||||
"you are assigned to issue hyperhive/hyperhive#7: router (https://forge/7)"
|
||||
);
|
||||
assert_eq!(got[0].op.source, "router:assigned");
|
||||
// The match gates: an unassign event reaches no assigned sink.
|
||||
assert!(
|
||||
flow("assigned")
|
||||
.evaluate("issue_assign", &event("unassigned", &["atlas"], &[], &[]))
|
||||
.is_empty()
|
||||
);
|
||||
// The source gates: another event type starts nothing.
|
||||
assert!(
|
||||
flow("assigned")
|
||||
.evaluate("issues", &event("assigned", &["atlas"], &[], &[]))
|
||||
.is_empty()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unassigning_rewrites_the_same_key_for_the_removed_agent() {
|
||||
let got = flow("unassigned").evaluate(
|
||||
"pull_request_assign",
|
||||
&event("unassigned", &[], &["argus"], &[]),
|
||||
);
|
||||
assert_eq!(got.len(), 1);
|
||||
assert_eq!(got[0].agent, "argus");
|
||||
assert_eq!(got[0].op.key, "forge:hyperhive/hyperhive#7:assignment");
|
||||
assert!(
|
||||
got[0]
|
||||
.op
|
||||
.summary
|
||||
.starts_with("you are not assigned to issue")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn closing_clears_by_prefix_for_every_holder() {
|
||||
let got =
|
||||
flow("closed").evaluate("issues", &event("closed", &[], &[], &["atlas", "argus"]));
|
||||
assert_eq!(got.len(), 2);
|
||||
for a in &got {
|
||||
assert_eq!(a.op.op, Op::Clear);
|
||||
assert!(a.op.prefix);
|
||||
assert_eq!(a.op.key, "forge:hyperhive/hyperhive#7:");
|
||||
assert_eq!(a.op.summary, "");
|
||||
}
|
||||
assert!(
|
||||
flow("closed")
|
||||
.evaluate("issues", &event("reopened", &[], &[], &["atlas"]))
|
||||
.is_empty()
|
||||
);
|
||||
}
|
||||
|
||||
/// A match node with two children runs both: the branching the tree
|
||||
/// allows.
|
||||
#[test]
|
||||
fn a_match_with_two_children_runs_both() {
|
||||
let mut f = flow("assigned");
|
||||
let NodeKind::Todo(sink) = f.nodes[2].kind.clone() else {
|
||||
panic!("the third node is the sink")
|
||||
};
|
||||
f.nodes.push(Node {
|
||||
id: "second".to_owned(),
|
||||
kind: NodeKind::Todo(TodoSink {
|
||||
to: "assignees_removed".to_owned(),
|
||||
..sink
|
||||
}),
|
||||
});
|
||||
f.edges.push(Edge {
|
||||
from: "match".to_owned(),
|
||||
to: "second".to_owned(),
|
||||
});
|
||||
assert_eq!(f.validate(), Ok(()));
|
||||
let got = f.evaluate(
|
||||
"issue_assign",
|
||||
&event("assigned", &["atlas"], &["argus"], &[]),
|
||||
);
|
||||
let mut nodes: Vec<(&str, &str)> = got
|
||||
.iter()
|
||||
.map(|a| (a.node.as_str(), a.agent.as_str()))
|
||||
.collect();
|
||||
nodes.sort_unstable();
|
||||
assert_eq!(nodes, [("second", "argus"), ("sink", "atlas")]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_flow_that_is_not_a_tree_is_refused() {
|
||||
let edge = |from: &str, to: &str| Edge {
|
||||
from: from.to_owned(),
|
||||
to: to.to_owned(),
|
||||
};
|
||||
let mut cycle = flow("assigned");
|
||||
cycle.edges[0] = edge("sink", "match");
|
||||
cycle.edges.push(edge("match", "sink"));
|
||||
assert!(cycle.validate().is_err(), "a cycle");
|
||||
|
||||
let mut orphan = flow("assigned");
|
||||
orphan.edges.pop();
|
||||
assert!(orphan.validate().is_err(), "an unreachable sink");
|
||||
|
||||
let mut two_parents = flow("assigned");
|
||||
two_parents.edges.push(edge("source", "sink"));
|
||||
assert!(two_parents.validate().is_err(), "a sink with two parents");
|
||||
|
||||
let mut sink_child = flow("assigned");
|
||||
sink_child.nodes.push(Node {
|
||||
id: "m2".to_owned(),
|
||||
kind: NodeKind::Match { when: Vec::new() },
|
||||
});
|
||||
sink_child.edges.push(edge("sink", "m2"));
|
||||
assert!(sink_child.validate().is_err(), "a sink with a child");
|
||||
|
||||
let mut dangling = flow("assigned");
|
||||
dangling.edges.push(edge("match", "nowhere"));
|
||||
assert!(dangling.validate().is_err(), "an edge to no node");
|
||||
|
||||
let mut two_sources = flow("assigned");
|
||||
two_sources.nodes[1].kind = NodeKind::Source { events: Vec::new() };
|
||||
assert!(two_sources.validate().is_err(), "two sources");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_placeholder_expands_to_its_field_and_an_unknown_one_stays() {
|
||||
let fields = event("assigned", &["a", "b"], &[], &[]);
|
||||
assert_eq!(
|
||||
expand("{repo}#{number} {assignees_added} {nope} {", &fields),
|
||||
"hyperhive/hyperhive#7 a, b {nope} {"
|
||||
);
|
||||
}
|
||||
}
|
||||
295
swarm-notify-router/src/main.rs
Normal file
295
swarm-notify-router/src/main.rs
Normal file
|
|
@ -0,0 +1,295 @@
|
|||
//! The swarm notification router.
|
||||
//!
|
||||
//! Drains its durable consumer on the forge events stream
|
||||
//! (`swarm_wire::forge_events`), runs every flow over each delivery
|
||||
//! ([`flow`], [`route`]), and queues the todo ops the flows emit on each
|
||||
//! recipient agent's subject of `ROUTER_TODOS` (`swarm_wire::router_todo`).
|
||||
//! Each agent's harness applies them to its own todo list.
|
||||
//!
|
||||
//! Per issue it keeps a snapshot in its state bucket ([`event::Snapshot`]):
|
||||
//! the assignees it last saw and the agents it queued a todo for. An event is
|
||||
//! acked once its ops are queued and the snapshot is written, so a crash
|
||||
//! between the two redelivers it, and the stream drops the republished ops
|
||||
//! by their message id.
|
||||
//!
|
||||
//! It connects to the queue as its own OIDC client, whose secret it reads
|
||||
//! from the store under its own certificate, as `swarm-claude-login` does.
|
||||
//! It holds no forge token and no webhook secret.
|
||||
|
||||
use std::collections::BTreeSet;
|
||||
use std::path::PathBuf;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use anyhow::{Context as _, Result, bail};
|
||||
use async_nats::jetstream::{self, consumer::pull, kv, message::PublishMessage};
|
||||
use futures_util::{StreamExt as _, TryStreamExt as _};
|
||||
use swarm_queue_client::{ClientSecret, QueueConfig};
|
||||
use swarm_secret_client::stored_secret::StoredSecret;
|
||||
use swarm_wire::{forge_events, notify_router, router_todo};
|
||||
|
||||
mod event;
|
||||
mod flow;
|
||||
mod route;
|
||||
mod settle;
|
||||
|
||||
/// The cert-auth role this service logs in to the store as. `swarm-bao.nix`
|
||||
/// writes it, matching the leaf's subject.
|
||||
const CERT_ROLE: &str = "swarm-notify-router";
|
||||
|
||||
/// The environment prefix `swarm-notify-router.nix` sets.
|
||||
const PREFIX: &str = "SWARM_NOTIFY_ROUTER";
|
||||
|
||||
/// How many events one pull asks for, and how long it waits for them.
|
||||
const BATCH: usize = 16;
|
||||
const PULL_WAIT: Duration = Duration::from_secs(30);
|
||||
|
||||
/// How long an event may stay unacked before the stream redelivers it.
|
||||
const EVENT_ACK_WAIT: Duration = Duration::from_mins(1);
|
||||
|
||||
/// How long the router waits after a failed pull or event before it pulls
|
||||
/// again, when the failed event comes back first.
|
||||
const RETRY_DELAY: Duration = Duration::from_secs(30);
|
||||
|
||||
/// How long the list of agents is reused before it is read again.
|
||||
const ROSTER_TTL: Duration = Duration::from_mins(1);
|
||||
|
||||
#[tokio::main]
|
||||
async fn main() -> Result<()> {
|
||||
hive_log::init();
|
||||
swarm_secret_client::install_crypto_provider();
|
||||
let var = |name: &str| {
|
||||
let full = format!("{PREFIX}_{name}");
|
||||
std::env::var(&full).with_context(|| format!("{full} is unset"))
|
||||
};
|
||||
let client_id = var("OIDC_CLIENT_ID")?;
|
||||
let secret =
|
||||
swarm_secret_client::read_with_startup_retry("reading the queue client secret", || {
|
||||
client_secret(&client_id)
|
||||
})
|
||||
.await?;
|
||||
let reread_id = client_id.clone();
|
||||
let secret = StoredSecret::new(client_id.clone(), secret, move || {
|
||||
let client_id = reread_id.clone();
|
||||
async move { client_secret(&client_id).await.map_err(Into::into) }
|
||||
});
|
||||
let queue = swarm_queue_client::connect(QueueConfig {
|
||||
url: var("NATS_URL")?,
|
||||
token_endpoint: var("OIDC_TOKEN_ENDPOINT")?,
|
||||
client_id,
|
||||
client_secret: ClientSecret::Stored(secret),
|
||||
ca_file: std::env::var(format!("{PREFIX}_OIDC_CA_FILE"))
|
||||
.ok()
|
||||
.map(PathBuf::from),
|
||||
})
|
||||
.await?;
|
||||
|
||||
let flows: Vec<flow::Flow> = flow::builtins()
|
||||
.into_iter()
|
||||
.filter(|f| match f.validate() {
|
||||
Ok(()) => true,
|
||||
Err(reason) => {
|
||||
tracing::error!(flow = %f.id, %reason, "notify router: skipping a flow that is not a tree");
|
||||
false
|
||||
}
|
||||
})
|
||||
.collect();
|
||||
let mut router = Router::open(&queue, flows).await?;
|
||||
tracing::info!(
|
||||
stream = forge_events::STREAM,
|
||||
"notify router: routing forge events"
|
||||
);
|
||||
loop {
|
||||
if let Err(e) = router.pull_once().await {
|
||||
tracing::warn!(
|
||||
error = format!("{e:#}"),
|
||||
"notify router: pulling events failed; retrying"
|
||||
);
|
||||
tokio::time::sleep(RETRY_DELAY).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
struct Router {
|
||||
js: jetstream::Context,
|
||||
events: jetstream::consumer::Consumer<pull::Config>,
|
||||
state: kv::Store,
|
||||
flows: Vec<flow::Flow>,
|
||||
roster: Option<(Instant, BTreeSet<String>)>,
|
||||
}
|
||||
|
||||
impl Router {
|
||||
async fn open(queue: &async_nats::Client, flows: Vec<flow::Flow>) -> Result<Self> {
|
||||
let js = jetstream::new(queue.clone());
|
||||
let events = swarm_queue_client::forge_events::open_or_create(queue)
|
||||
.await
|
||||
.context("opening the forge events stream")?;
|
||||
swarm_queue_client::router_todo::open_or_create(queue)
|
||||
.await
|
||||
.context("opening the router todo stream")?;
|
||||
let events = events
|
||||
.create_consumer(pull::Config {
|
||||
durable_name: Some(forge_events::ROUTER_CONSUMER.to_owned()),
|
||||
description: Some("The notification router".to_owned()),
|
||||
filter_subject: forge_events::subject_wildcard(),
|
||||
ack_wait: EVENT_ACK_WAIT,
|
||||
..Default::default()
|
||||
})
|
||||
.await
|
||||
.context("creating the router's consumer")?;
|
||||
let state = match js.get_key_value(notify_router::STATE_BUCKET).await {
|
||||
Ok(store) => store,
|
||||
Err(e) => {
|
||||
tracing::info!(reason = %e, "notify router: state bucket not available, creating it");
|
||||
js.create_key_value(kv::Config {
|
||||
bucket: notify_router::STATE_BUCKET.to_owned(),
|
||||
description: "Per-issue assignees and todo holders".to_owned(),
|
||||
history: 1,
|
||||
..Default::default()
|
||||
})
|
||||
.await
|
||||
.context("creating the state bucket")?
|
||||
}
|
||||
};
|
||||
Ok(Self {
|
||||
js,
|
||||
events,
|
||||
state,
|
||||
flows,
|
||||
roster: None,
|
||||
})
|
||||
}
|
||||
|
||||
/// Pull one batch and handle each event in stream order.
|
||||
async fn pull_once(&mut self) -> Result<()> {
|
||||
let batch = self
|
||||
.events
|
||||
.batch()
|
||||
.max_messages(BATCH)
|
||||
.expires(PULL_WAIT)
|
||||
.messages()
|
||||
.await
|
||||
.context("requesting a batch")?;
|
||||
let batch = batch.map(|m| m.map_err(|e| anyhow::anyhow!("{e}")));
|
||||
settle::batch(batch, async |m: &jetstream::Message| self.handle(m).await).await
|
||||
}
|
||||
|
||||
async fn handle(&mut self, message: &jetstream::Message) -> Result<(), Skip> {
|
||||
let event_type = forge_events::event_type(&message.subject)
|
||||
.ok_or_else(|| Skip::Unusable("not a forge event subject".to_owned()))?
|
||||
.to_owned();
|
||||
let Some(delivery) = event::Delivery::parse(&message.payload) else {
|
||||
// A delivery that names no issue or PR has nothing to route.
|
||||
return Ok(());
|
||||
};
|
||||
let seq = message
|
||||
.info()
|
||||
.map(|i| i.stream_sequence)
|
||||
.map_err(|e| Skip::Unusable(format!("no stream sequence: {e}")))?;
|
||||
let key = delivery.state_key();
|
||||
let snapshot: event::Snapshot = match self.state.get(&key).await.map_err(retry)? {
|
||||
Some(bytes) => serde_json::from_slice(&bytes).unwrap_or_else(|e| {
|
||||
tracing::warn!(%key, error = %e, "notify router: unreadable snapshot; starting afresh");
|
||||
event::Snapshot::default()
|
||||
}),
|
||||
None => event::Snapshot::default(),
|
||||
};
|
||||
let roster = self.roster().await.map_err(Skip::Retry)?;
|
||||
let routed = route::route(&event_type, &delivery, &snapshot, &self.flows, |a| {
|
||||
roster.contains(a)
|
||||
});
|
||||
for action in &routed.actions {
|
||||
self.queue(seq, action).await.map_err(Skip::Retry)?;
|
||||
}
|
||||
if !routed.skipped.is_empty() {
|
||||
tracing::info!(%event_type, repo = %delivery.repo, number = delivery.number, skipped = ?routed.skipped, "notify router: recipients that are not agents got nothing");
|
||||
}
|
||||
if routed.next != snapshot {
|
||||
let bytes = serde_json::to_vec(&routed.next).expect("a snapshot encodes");
|
||||
self.state.put(&key, bytes.into()).await.map_err(retry)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Queue one op on its agent's subject and wait for the stream's ack.
|
||||
async fn queue(&self, seq: u64, action: &flow::Action) -> Result<()> {
|
||||
let subject = router_todo::subject(&action.agent);
|
||||
let body = serde_json::to_vec(&action.op).expect("an op encodes");
|
||||
let id = format!("{seq}:{}:{}:{}", action.flow, action.node, action.agent);
|
||||
self.js
|
||||
.send_publish(
|
||||
subject.clone(),
|
||||
PublishMessage::build().payload(body.into()).message_id(id),
|
||||
)
|
||||
.await
|
||||
.with_context(|| format!("publishing to {subject}"))?
|
||||
.await
|
||||
.with_context(|| format!("waiting for {subject}'s ack"))?;
|
||||
tracing::info!(agent = %action.agent, flow = %action.flow, key = %action.op.key, op = ?action.op.op, "notify router: todo op queued");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// The agents of this swarm: the agent names in the agent-status bucket's
|
||||
/// keys (`<hive>.<agent>`), read from the stream's subject list.
|
||||
async fn roster(&mut self) -> Result<BTreeSet<String>> {
|
||||
if let Some((at, agents)) = &self.roster
|
||||
&& at.elapsed() < ROSTER_TTL
|
||||
{
|
||||
return Ok(agents.clone());
|
||||
}
|
||||
let bucket = swarm_wire::agent_status::BUCKET;
|
||||
let stream = self
|
||||
.js
|
||||
.get_stream(format!("KV_{bucket}"))
|
||||
.await
|
||||
.context("opening the agent-status bucket")?;
|
||||
let subjects: Vec<(String, usize)> = stream
|
||||
.info_with_subjects(format!("$KV.{bucket}.>"))
|
||||
.await
|
||||
.context("listing agent-status keys")?
|
||||
.try_collect()
|
||||
.await
|
||||
.context("listing agent-status keys")?;
|
||||
let prefix = format!("$KV.{bucket}.");
|
||||
let agents: BTreeSet<String> = subjects
|
||||
.into_iter()
|
||||
.filter_map(|(subject, _)| {
|
||||
let key = subject.strip_prefix(&prefix)?;
|
||||
swarm_wire::agent_status::split_key(key).map(|(_, agent)| agent.to_owned())
|
||||
})
|
||||
.collect();
|
||||
self.roster = Some((Instant::now(), agents.clone()));
|
||||
Ok(agents)
|
||||
}
|
||||
}
|
||||
|
||||
/// Why an event was not handled.
|
||||
enum Skip {
|
||||
/// It can never be handled; dropped for good.
|
||||
Unusable(String),
|
||||
/// The queue or the store failed; tried again.
|
||||
Retry(anyhow::Error),
|
||||
}
|
||||
|
||||
fn retry(e: impl std::error::Error + Send + Sync + 'static) -> Skip {
|
||||
Skip::Retry(e.into())
|
||||
}
|
||||
|
||||
async fn client_secret(client_id: &str) -> Result<String> {
|
||||
use swarm_secret_client::builder::{ClientSecret, client_secret_path};
|
||||
|
||||
let path = client_secret_path(client_id)?;
|
||||
let store = swarm_secret_client::SecretStore::from_env(CERT_ROLE)
|
||||
.await
|
||||
.context("logging in to the secret store")?;
|
||||
let stored: Option<ClientSecret> = store
|
||||
.read_optional(&path)
|
||||
.await
|
||||
.with_context(|| format!("reading {path}"))?;
|
||||
match stored {
|
||||
Some(s) if !s.value.trim().is_empty() => Ok(s.value.trim().to_owned()),
|
||||
_ => bail!(
|
||||
"nothing at {path}: swarm-secret-publish, in authelia's container, has not \
|
||||
published this service's client secret"
|
||||
),
|
||||
}
|
||||
}
|
||||
194
swarm-notify-router/src/route.rs
Normal file
194
swarm-notify-router/src/route.rs
Normal file
|
|
@ -0,0 +1,194 @@
|
|||
//! One event through every flow: the ops to queue and the issue's next
|
||||
//! snapshot. No I/O, so the whole decision is testable.
|
||||
|
||||
use std::collections::BTreeSet;
|
||||
|
||||
use swarm_wire::router_todo::Op;
|
||||
|
||||
use crate::event::{Delivery, Snapshot, fields, is_assign};
|
||||
use crate::flow::{Action, Flow};
|
||||
|
||||
/// What one event comes to.
|
||||
#[derive(Debug, PartialEq, Eq)]
|
||||
pub struct Routed {
|
||||
/// The ops to queue, one per agent that exists.
|
||||
pub actions: Vec<Action>,
|
||||
/// Recipients that are not agents of this swarm, so got nothing.
|
||||
pub skipped: Vec<String>,
|
||||
pub next: Snapshot,
|
||||
}
|
||||
|
||||
/// Run `flows` over one delivery of `event_type` on an issue whose snapshot is
|
||||
/// `snapshot`. `is_agent` says whether a recipient is an agent of this swarm.
|
||||
pub fn route(
|
||||
event_type: &str,
|
||||
delivery: &Delivery,
|
||||
snapshot: &Snapshot,
|
||||
flows: &[Flow],
|
||||
is_agent: impl Fn(&str) -> bool,
|
||||
) -> Routed {
|
||||
let fields = fields(event_type, delivery, snapshot);
|
||||
let mut actions = Vec::new();
|
||||
let mut skipped = Vec::new();
|
||||
for action in flows.iter().flat_map(|f| f.evaluate(event_type, &fields)) {
|
||||
if is_agent(&action.agent) {
|
||||
actions.push(action);
|
||||
} else if !skipped.contains(&action.agent) {
|
||||
skipped.push(action.agent);
|
||||
}
|
||||
}
|
||||
let mut next = snapshot.clone();
|
||||
if is_assign(event_type) {
|
||||
// A skipped assignee is left out of the snapshot, so the next assign
|
||||
// event counts it as added again: an agent assigned before it is in
|
||||
// the roster gets its todo once it is.
|
||||
let mut assignees = delivery.assignees.clone();
|
||||
assignees.retain(|a| !skipped.contains(a));
|
||||
next.assignees = Some(assignees);
|
||||
}
|
||||
let mut cleared = BTreeSet::new();
|
||||
for action in &actions {
|
||||
match action.op.op {
|
||||
Op::Upsert => {
|
||||
next.holders.insert(action.agent.clone());
|
||||
}
|
||||
Op::Clear => {
|
||||
cleared.insert(action.agent.clone());
|
||||
}
|
||||
}
|
||||
}
|
||||
next.holders.retain(|h| !cleared.contains(h));
|
||||
Routed {
|
||||
actions,
|
||||
skipped,
|
||||
next,
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::flow::builtins;
|
||||
|
||||
fn delivery(action: &str, assignees: &[&str]) -> Delivery {
|
||||
Delivery {
|
||||
action: action.to_owned(),
|
||||
repo: "o/r".to_owned(),
|
||||
repo_id: 1,
|
||||
number: 9,
|
||||
title: "t".to_owned(),
|
||||
url: "u".to_owned(),
|
||||
sender: "mara".to_owned(),
|
||||
is_pull: false,
|
||||
assignees: assignees.iter().map(|a| (*a).to_owned()).collect(),
|
||||
}
|
||||
}
|
||||
|
||||
fn agents(a: &Routed) -> Vec<(&str, Op)> {
|
||||
a.actions
|
||||
.iter()
|
||||
.map(|x| (x.agent.as_str(), x.op.op))
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Assign, reassign, close: the three built-ins in the order an issue
|
||||
/// meets them, carrying the snapshot from one event to the next.
|
||||
#[test]
|
||||
fn an_issue_assigned_reassigned_and_closed_queues_and_clears_its_todos() {
|
||||
let flows = builtins();
|
||||
let is_agent = |a: &str| a != "mara";
|
||||
|
||||
let one = route(
|
||||
"issue_assign",
|
||||
&delivery("assigned", &["atlas", "mara"]),
|
||||
&Snapshot::default(),
|
||||
&flows,
|
||||
is_agent,
|
||||
);
|
||||
assert_eq!(agents(&one), [("atlas", Op::Upsert)]);
|
||||
assert_eq!(one.skipped, ["mara"]);
|
||||
assert_eq!(one.next.holders, BTreeSet::from(["atlas".to_owned()]));
|
||||
|
||||
let two = route(
|
||||
"issue_assign",
|
||||
&delivery("unassigned", &["mara"]),
|
||||
&one.next,
|
||||
&flows,
|
||||
is_agent,
|
||||
);
|
||||
assert_eq!(agents(&two), [("atlas", Op::Upsert)]);
|
||||
assert!(two.actions[0].op.summary.contains("not assigned"));
|
||||
|
||||
let three = route(
|
||||
"issue_assign",
|
||||
&delivery("assigned", &["mara", "iris"]),
|
||||
&two.next,
|
||||
&flows,
|
||||
is_agent,
|
||||
);
|
||||
assert_eq!(agents(&three), [("iris", Op::Upsert)]);
|
||||
|
||||
let closed = route(
|
||||
"issues",
|
||||
&delivery("closed", &["mara", "iris"]),
|
||||
&three.next,
|
||||
&flows,
|
||||
is_agent,
|
||||
);
|
||||
let mut got = agents(&closed);
|
||||
got.sort_unstable_by_key(|(a, _)| *a);
|
||||
assert_eq!(got, [("atlas", Op::Clear), ("iris", Op::Clear)]);
|
||||
assert!(closed.next.holders.is_empty());
|
||||
// A close leaves the assignee snapshot as the last assign set it.
|
||||
assert_eq!(closed.next.assignees, three.next.assignees);
|
||||
}
|
||||
|
||||
/// An agent assigned before it reports status gets nothing then, and its
|
||||
/// todo on the next assign event, once it is in the roster.
|
||||
#[test]
|
||||
fn an_assignee_not_yet_in_the_roster_gets_its_todo_on_the_next_assign() {
|
||||
let flows = builtins();
|
||||
let early = route(
|
||||
"issue_assign",
|
||||
&delivery("assigned", &["nova"]),
|
||||
&Snapshot::default(),
|
||||
&flows,
|
||||
|_| false,
|
||||
);
|
||||
assert!(early.actions.is_empty());
|
||||
assert_eq!(early.skipped, ["nova"]);
|
||||
assert_eq!(early.next.assignees, Some(BTreeSet::new()));
|
||||
|
||||
let later = route(
|
||||
"issue_assign",
|
||||
&delivery("assigned", &["nova", "atlas"]),
|
||||
&early.next,
|
||||
&flows,
|
||||
|_| true,
|
||||
);
|
||||
let mut got = agents(&later);
|
||||
got.sort_unstable_by_key(|(a, _)| *a);
|
||||
assert_eq!(got, [("atlas", Op::Upsert), ("nova", Op::Upsert)]);
|
||||
assert_eq!(
|
||||
later.next.assignees,
|
||||
Some(BTreeSet::from(["atlas".to_owned(), "nova".to_owned()]))
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_event_no_flow_matches_changes_nothing() {
|
||||
let before = Snapshot {
|
||||
assignees: Some(BTreeSet::from(["atlas".to_owned()])),
|
||||
holders: BTreeSet::from(["atlas".to_owned()]),
|
||||
};
|
||||
let r = route(
|
||||
"issue_comment",
|
||||
&delivery("created", &[]),
|
||||
&before,
|
||||
&builtins(),
|
||||
|_| true,
|
||||
);
|
||||
assert!(r.actions.is_empty());
|
||||
assert_eq!(r.next, before);
|
||||
}
|
||||
}
|
||||
155
swarm-notify-router/src/settle.rs
Normal file
155
swarm-notify-router/src/settle.rs
Normal file
|
|
@ -0,0 +1,155 @@
|
|||
//! Handling one pulled batch in stream order.
|
||||
//!
|
||||
//! An issue's snapshot diff depends on the order its events are handled in,
|
||||
//! so a failed event must not let a later one overtake it. After the first
|
||||
//! event that has to be retried, nothing more of the batch is handled: it and
|
||||
//! every later event are nakked for immediate redelivery, and the caller waits
|
||||
//! before it pulls again, which hands them back in stream order.
|
||||
|
||||
use anyhow::Result;
|
||||
use async_nats::jetstream::AckKind;
|
||||
use futures_util::{Stream, StreamExt as _};
|
||||
|
||||
use crate::Skip;
|
||||
|
||||
/// One pulled event: what it is called in the log, and how it is acked.
|
||||
pub(crate) trait Pulled {
|
||||
fn subject(&self) -> &str;
|
||||
async fn settle(&self, ack: AckKind) -> Result<()>;
|
||||
}
|
||||
|
||||
impl Pulled for async_nats::jetstream::Message {
|
||||
fn subject(&self) -> &str {
|
||||
&self.subject
|
||||
}
|
||||
|
||||
async fn settle(&self, ack: AckKind) -> Result<()> {
|
||||
self.ack_with(ack)
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!("acking an event: {e}"))
|
||||
}
|
||||
}
|
||||
|
||||
/// Handle each event of `batch` with `handle` and settle it. Returns the
|
||||
/// error of the first event that has to be retried, once the rest of the
|
||||
/// batch is nakked behind it.
|
||||
pub(crate) async fn batch<P: Pulled>(
|
||||
batch: impl Stream<Item = Result<P>>,
|
||||
mut handle: impl AsyncFnMut(&P) -> Result<(), Skip>,
|
||||
) -> Result<()> {
|
||||
let mut batch = std::pin::pin!(batch);
|
||||
let mut failed: Option<anyhow::Error> = None;
|
||||
while let Some(event) = batch.next().await {
|
||||
let event = event?;
|
||||
let ack = if failed.is_some() {
|
||||
AckKind::Nak(None)
|
||||
} else {
|
||||
match handle(&event).await {
|
||||
Ok(()) => AckKind::Ack,
|
||||
Err(Skip::Unusable(reason)) => {
|
||||
tracing::warn!(subject = event.subject(), %reason, "notify router: dropping an unusable event");
|
||||
AckKind::Term
|
||||
}
|
||||
Err(Skip::Retry(e)) => {
|
||||
failed = Some(e);
|
||||
AckKind::Nak(None)
|
||||
}
|
||||
}
|
||||
};
|
||||
event.settle(ack).await?;
|
||||
}
|
||||
match failed {
|
||||
Some(e) => Err(e.context(
|
||||
"an event failed; it and the rest of its batch are redelivered in stream order",
|
||||
)),
|
||||
None => Ok(()),
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use std::cell::RefCell;
|
||||
|
||||
use super::*;
|
||||
|
||||
struct Fake<'a> {
|
||||
name: &'static str,
|
||||
settled: &'a RefCell<Vec<(&'static str, String)>>,
|
||||
}
|
||||
|
||||
impl Pulled for Fake<'_> {
|
||||
fn subject(&self) -> &str {
|
||||
self.name
|
||||
}
|
||||
|
||||
async fn settle(&self, ack: AckKind) -> Result<()> {
|
||||
self.settled
|
||||
.borrow_mut()
|
||||
.push((self.name, format!("{ack:?}")));
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
/// `assigned` fails, so `unassigned` after it is not handled: both come
|
||||
/// back, in order, rather than the unassign overtaking the assign.
|
||||
#[tokio::test]
|
||||
async fn after_a_retry_the_rest_of_the_batch_is_not_handled_and_comes_back() {
|
||||
let settled = RefCell::new(Vec::new());
|
||||
let events = ["opened", "assigned", "unassigned", "closed"].map(|name| {
|
||||
Ok(Fake {
|
||||
name,
|
||||
settled: &settled,
|
||||
})
|
||||
});
|
||||
let mut handled = Vec::new();
|
||||
let result = batch(futures_util::stream::iter(events), async |e: &Fake<'_>| {
|
||||
handled.push(e.name);
|
||||
if e.name == "assigned" {
|
||||
Err(Skip::Retry(anyhow::anyhow!("the queue is down")))
|
||||
} else {
|
||||
Ok(())
|
||||
}
|
||||
})
|
||||
.await;
|
||||
assert!(result.is_err());
|
||||
assert_eq!(handled, ["opened", "assigned"]);
|
||||
let nak = format!("{:?}", AckKind::Nak(None));
|
||||
assert_eq!(
|
||||
settled.into_inner(),
|
||||
[
|
||||
("opened", format!("{:?}", AckKind::Ack)),
|
||||
("assigned", nak.clone()),
|
||||
("unassigned", nak.clone()),
|
||||
("closed", nak),
|
||||
]
|
||||
);
|
||||
}
|
||||
|
||||
/// Control: an unusable event is dropped and the batch goes on.
|
||||
#[tokio::test]
|
||||
async fn an_unusable_event_is_dropped_and_the_batch_goes_on() {
|
||||
let settled = RefCell::new(Vec::new());
|
||||
let events = ["bad", "good"].map(|name| {
|
||||
Ok(Fake {
|
||||
name,
|
||||
settled: &settled,
|
||||
})
|
||||
});
|
||||
let result = batch(futures_util::stream::iter(events), async |e: &Fake<'_>| {
|
||||
if e.name == "bad" {
|
||||
Err(Skip::Unusable("no".to_owned()))
|
||||
} else {
|
||||
Ok(())
|
||||
}
|
||||
})
|
||||
.await;
|
||||
assert!(result.is_ok());
|
||||
assert_eq!(
|
||||
settled.into_inner(),
|
||||
[
|
||||
("bad", format!("{:?}", AckKind::Term)),
|
||||
("good", format!("{:?}", AckKind::Ack)),
|
||||
]
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
@ -63,6 +63,14 @@ agent-archetype = ["async-nats/jetstream"]
|
|||
# initial `AGENTS.md` text with one direct get. Explicit for the same reason as
|
||||
# `notices`.
|
||||
agents-md-initial = ["async-nats/jetstream"]
|
||||
# `forge_events::open_or_create`, for swarm-controller, which publishes verified
|
||||
# forge deliveries, and the notification router, which reads them. Explicit
|
||||
# for the same reason as `notices`.
|
||||
forge-events = ["async-nats/jetstream"]
|
||||
# `router_todo`, for the notification router, which creates the stream, and
|
||||
# the agent harness, which drains its own consumer. Explicit for the same
|
||||
# reason as `notices`.
|
||||
router-todo = ["async-nats/jetstream"]
|
||||
|
||||
[dependencies]
|
||||
# Bare (no `kv`/`jetstream`) unless a consumer opts into the `kv` feature
|
||||
|
|
@ -97,6 +105,10 @@ tracing.workspace = true
|
|||
# A loopback stand-in token endpoint for the client secret re-read tests.
|
||||
axum.workspace = true
|
||||
data-encoding.workspace = true
|
||||
# The ports-file directory of the throwaway `nats-server` the stream tests run.
|
||||
tempfile = "3"
|
||||
# StreamExt::next on a pulled batch in those tests.
|
||||
futures-util.workspace = true
|
||||
|
||||
[lints]
|
||||
workspace = true
|
||||
|
|
|
|||
77
swarm-queue-client/src/forge_events.rs
Normal file
77
swarm-queue-client/src/forge_events.rs
Normal file
|
|
@ -0,0 +1,77 @@
|
|||
//! Opening the forge events stream. The name, subjects and why it exists are
|
||||
//! `swarm_wire::forge_events`.
|
||||
|
||||
use swarm_wire::forge_events::{MAX_AGE, STREAM, subject_wildcard};
|
||||
|
||||
/// The config [`open_or_create`] creates the stream with.
|
||||
fn stream_config() -> async_nats::jetstream::stream::Config {
|
||||
async_nats::jetstream::stream::Config {
|
||||
name: STREAM.to_owned(),
|
||||
description: Some("Verified forge webhook deliveries".to_owned()),
|
||||
subjects: vec![subject_wildcard()],
|
||||
max_age: MAX_AGE,
|
||||
// A redelivered webhook carries the same `X-Forgejo-Delivery`, which
|
||||
// the controller sends as the message id.
|
||||
duplicate_window: std::time::Duration::from_hours(1),
|
||||
..Default::default()
|
||||
}
|
||||
}
|
||||
|
||||
/// Open the stream, creating it if nothing has yet. Called by the controller,
|
||||
/// which publishes into it, and by the router, which reads it: either may come
|
||||
/// up first.
|
||||
pub async fn open_or_create(
|
||||
client: &async_nats::Client,
|
||||
) -> Result<async_nats::jetstream::stream::Stream, crate::Error> {
|
||||
let js = async_nats::jetstream::new(client.clone());
|
||||
match js.get_stream(STREAM).await {
|
||||
Ok(stream) => Ok(stream),
|
||||
Err(e) => {
|
||||
tracing::info!(stream = STREAM, reason = %e, "forge events stream not available, creating it");
|
||||
js.create_stream(stream_config())
|
||||
.await
|
||||
.map_err(|source| crate::Error::CreateStream {
|
||||
stream: STREAM,
|
||||
source,
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use swarm_wire::forge_events::{STREAM, subject};
|
||||
|
||||
use super::open_or_create;
|
||||
use crate::test_nats::TestNats;
|
||||
|
||||
/// Either end may create the stream first, and a redelivered webhook is
|
||||
/// kept once.
|
||||
#[tokio::test]
|
||||
async fn the_stream_is_created_once_and_keeps_a_redelivery_once() {
|
||||
let nats = TestNats::start();
|
||||
let client = nats.client().await;
|
||||
open_or_create(&client).await.expect("created");
|
||||
open_or_create(&client).await.expect("opened as it is");
|
||||
let js = async_nats::jetstream::new(client);
|
||||
for (event, id) in [
|
||||
("issues", "d-1"),
|
||||
("issues", "d-1"),
|
||||
("issue_assign", "d-2"),
|
||||
] {
|
||||
js.send_publish(
|
||||
subject(event),
|
||||
async_nats::jetstream::message::PublishMessage::build()
|
||||
.payload("{}".into())
|
||||
.message_id(id),
|
||||
)
|
||||
.await
|
||||
.expect("published")
|
||||
.await
|
||||
.expect("acked");
|
||||
}
|
||||
let mut stream = js.get_stream(STREAM).await.expect("the stream");
|
||||
let info = stream.info().await.expect("info");
|
||||
assert_eq!(info.state.messages, 2);
|
||||
}
|
||||
}
|
||||
|
|
@ -130,7 +130,7 @@ pub enum Error {
|
|||
source: async_nats::jetstream::kv::WatchError,
|
||||
},
|
||||
|
||||
#[cfg(feature = "notices")]
|
||||
#[cfg(any(feature = "notices", feature = "forge-events", feature = "router-todo"))]
|
||||
#[error("creating the {stream} stream")]
|
||||
CreateStream {
|
||||
stream: &'static str,
|
||||
|
|
@ -464,6 +464,20 @@ pub mod agent_term;
|
|||
#[cfg(feature = "subagent-term")]
|
||||
pub mod subagent_term;
|
||||
|
||||
/// Opening the stream swarm-controller keeps verified forge deliveries in.
|
||||
/// The names are `swarm_wire::forge_events`. Behind the `forge-events` feature.
|
||||
#[cfg(feature = "forge-events")]
|
||||
pub mod forge_events;
|
||||
|
||||
/// The notification router's todo queue: the stream, and each agent's own
|
||||
/// consumer. The names are `swarm_wire::router_todo`. Behind the
|
||||
/// `router-todo` feature.
|
||||
#[cfg(feature = "router-todo")]
|
||||
pub mod router_todo;
|
||||
|
||||
#[cfg(all(test, any(feature = "forge-events", feature = "router-todo")))]
|
||||
mod test_nats;
|
||||
|
||||
/// Only the fields this needs; authelia returns several.
|
||||
#[derive(serde::Deserialize)]
|
||||
struct TokenResponse {
|
||||
|
|
|
|||
203
swarm-queue-client/src/router_todo.rs
Normal file
203
swarm-queue-client/src/router_todo.rs
Normal file
|
|
@ -0,0 +1,203 @@
|
|||
//! The router's todo queue from both ends: the router creates the stream, and
|
||||
//! each agent's harness creates and drains its own consumer. The names,
|
||||
//! subjects and op are `swarm_wire::router_todo`.
|
||||
|
||||
use std::time::Duration;
|
||||
|
||||
use swarm_wire::router_todo::{
|
||||
DUPLICATE_WINDOW, MAX_AGE, MAX_MSGS_PER_SUBJECT, STREAM, consumer_name, subject,
|
||||
subject_wildcard,
|
||||
};
|
||||
|
||||
/// How long a delivered op may stay unacked before the server delivers it
|
||||
/// again.
|
||||
const ACK_WAIT: Duration = Duration::from_secs(30);
|
||||
|
||||
/// The config [`open_or_create`] creates the stream with.
|
||||
///
|
||||
/// Work-queue retention: an op leaves the stream once its agent acks it. The
|
||||
/// server then requires every consumer's filter to be disjoint from every
|
||||
/// other's, which one consumer per agent subject is.
|
||||
fn stream_config() -> async_nats::jetstream::stream::Config {
|
||||
use async_nats::jetstream::stream::{Config, DiscardPolicy, RetentionPolicy};
|
||||
|
||||
Config {
|
||||
name: STREAM.to_owned(),
|
||||
description: Some("Notification router todo ops, one subject per agent".to_owned()),
|
||||
subjects: vec![subject_wildcard()],
|
||||
retention: RetentionPolicy::WorkQueue,
|
||||
max_age: MAX_AGE,
|
||||
max_messages_per_subject: MAX_MSGS_PER_SUBJECT,
|
||||
discard: DiscardPolicy::Old,
|
||||
duplicate_window: DUPLICATE_WINDOW,
|
||||
..Default::default()
|
||||
}
|
||||
}
|
||||
|
||||
/// Open the stream, creating it if it does not exist yet. The router is its
|
||||
/// one writer and its one creator.
|
||||
pub async fn open_or_create(
|
||||
client: &async_nats::Client,
|
||||
) -> Result<async_nats::jetstream::stream::Stream, crate::Error> {
|
||||
let js = async_nats::jetstream::new(client.clone());
|
||||
match js.get_stream(STREAM).await {
|
||||
Ok(stream) => Ok(stream),
|
||||
Err(e) => {
|
||||
tracing::info!(stream = STREAM, reason = %e, "router todo stream not available, creating it");
|
||||
js.create_stream(stream_config())
|
||||
.await
|
||||
.map_err(|source| crate::Error::CreateStream {
|
||||
stream: STREAM,
|
||||
source,
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The durable pull consumer `agent`'s harness drains: its own subject, from
|
||||
/// the oldest op still queued, each acked explicitly.
|
||||
fn consumer_config(agent: &str) -> async_nats::jetstream::consumer::pull::Config {
|
||||
use async_nats::jetstream::consumer::{AckPolicy, DeliverPolicy, pull::Config};
|
||||
|
||||
Config {
|
||||
durable_name: Some(consumer_name(agent)),
|
||||
description: Some(format!("Router todo ops for {agent}")),
|
||||
filter_subject: subject(agent),
|
||||
deliver_policy: DeliverPolicy::All,
|
||||
ack_policy: AckPolicy::Explicit,
|
||||
ack_wait: ACK_WAIT,
|
||||
..Default::default()
|
||||
}
|
||||
}
|
||||
|
||||
/// Create `agent`'s consumer, or return it as it is when it already exists
|
||||
/// with the same config.
|
||||
///
|
||||
/// Created on the stream by name, without looking the stream up first: the
|
||||
/// agent's grant has no `STREAM.INFO`. The create subject is the extended
|
||||
/// form ending in the consumer's filter, the one subject the grant names.
|
||||
pub async fn open_own(
|
||||
client: &async_nats::Client,
|
||||
agent: &str,
|
||||
) -> Result<
|
||||
async_nats::jetstream::consumer::Consumer<async_nats::jetstream::consumer::pull::Config>,
|
||||
async_nats::jetstream::stream::ConsumerError,
|
||||
> {
|
||||
async_nats::jetstream::new(client.clone())
|
||||
.create_consumer_on_stream(consumer_config(agent), STREAM)
|
||||
.await
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use futures_util::StreamExt as _;
|
||||
use swarm_wire::router_todo::{STREAM, consumer_name, subject};
|
||||
|
||||
use super::{consumer_config, open_or_create, open_own};
|
||||
use crate::test_nats::TestNats;
|
||||
|
||||
async fn queue(js: &async_nats::jetstream::Context, agent: &str, id: &str) {
|
||||
js.send_publish(
|
||||
subject(agent),
|
||||
async_nats::jetstream::message::PublishMessage::build()
|
||||
.payload(id.as_bytes().to_vec().into())
|
||||
.message_id(id),
|
||||
)
|
||||
.await
|
||||
.expect("published")
|
||||
.await
|
||||
.expect("acked by the stream");
|
||||
}
|
||||
|
||||
async fn queued(js: &async_nats::jetstream::Context) -> u64 {
|
||||
js.get_stream(STREAM)
|
||||
.await
|
||||
.expect("the stream")
|
||||
.info()
|
||||
.await
|
||||
.expect("its info")
|
||||
.state
|
||||
.messages
|
||||
}
|
||||
|
||||
/// The server takes the stream and both agents' consumers, delivers each
|
||||
/// agent its own ops only, drops a republished op by its id, and removes an
|
||||
/// op once its agent acks it.
|
||||
#[tokio::test]
|
||||
async fn an_agents_consumer_drains_its_own_ops_and_an_ack_removes_them() {
|
||||
let nats = TestNats::start();
|
||||
let client = nats.client().await;
|
||||
let js = async_nats::jetstream::new(client.clone());
|
||||
open_or_create(&client).await.expect("created");
|
||||
open_or_create(&client).await.expect("opened as it is");
|
||||
|
||||
queue(&js, "atlas", "1:assigned:sink:atlas").await;
|
||||
queue(&js, "atlas", "1:assigned:sink:atlas").await;
|
||||
queue(&js, "argus", "1:assigned:sink:argus").await;
|
||||
assert_eq!(queued(&js).await, 2, "the republished op is dropped");
|
||||
|
||||
let atlas = open_own(&client, "atlas").await.expect("atlas's consumer");
|
||||
open_own(&client, "atlas")
|
||||
.await
|
||||
.expect("created again, unchanged");
|
||||
open_own(&client, "argus")
|
||||
.await
|
||||
.expect("argus's, disjoint from atlas's");
|
||||
|
||||
let mut batch = atlas
|
||||
.fetch()
|
||||
.max_messages(10)
|
||||
.messages()
|
||||
.await
|
||||
.expect("a batch");
|
||||
let mut got = Vec::new();
|
||||
while let Some(message) = batch.next().await {
|
||||
let message = message.expect("a message");
|
||||
got.push(message.subject.to_string());
|
||||
message.double_ack().await.expect("acked");
|
||||
}
|
||||
assert_eq!(got, [subject("atlas")]);
|
||||
assert_eq!(queued(&js).await, 1, "argus's op waits for argus");
|
||||
}
|
||||
|
||||
/// The grant's premise: in the extended create form the server refuses a
|
||||
/// payload filter that is not the subject's, so atlas's create subject
|
||||
/// cannot make a consumer that delivers argus's ops.
|
||||
#[tokio::test]
|
||||
async fn the_server_refuses_a_payload_filter_other_than_the_create_subjects() {
|
||||
let nats = TestNats::start();
|
||||
let client = nats.client().await;
|
||||
open_or_create(&client).await.expect("created");
|
||||
let create = |filter: String| {
|
||||
let mut config = consumer_config("atlas");
|
||||
config.filter_subject = filter;
|
||||
serde_json::to_vec(&serde_json::json!({
|
||||
"stream_name": STREAM,
|
||||
"config": async_nats::jetstream::consumer::IntoConsumerConfig::into_consumer_config(config),
|
||||
"action": "",
|
||||
}))
|
||||
.expect("encodes")
|
||||
};
|
||||
let subject_for_atlas = format!(
|
||||
"$JS.API.CONSUMER.CREATE.{STREAM}.{}.{}",
|
||||
consumer_name("atlas"),
|
||||
subject("atlas")
|
||||
);
|
||||
let answer = |body: Vec<u8>| {
|
||||
let client = client.clone();
|
||||
let subject_for_atlas = subject_for_atlas.clone();
|
||||
async move {
|
||||
let reply = client
|
||||
.request(subject_for_atlas, body.into())
|
||||
.await
|
||||
.expect("an answer");
|
||||
serde_json::from_slice::<serde_json::Value>(&reply.payload).expect("json")
|
||||
}
|
||||
};
|
||||
let refused = answer(create(subject("argus"))).await;
|
||||
assert!(refused.get("error").is_some(), "{refused}");
|
||||
// Control: the matching filter is accepted on the same subject.
|
||||
let accepted = answer(create(subject("atlas"))).await;
|
||||
assert!(accepted.get("error").is_none(), "{accepted}");
|
||||
}
|
||||
}
|
||||
78
swarm-queue-client/src/test_nats.rs
Normal file
78
swarm-queue-client/src/test_nats.rs
Normal file
|
|
@ -0,0 +1,78 @@
|
|||
//! A throwaway `nats-server` with `JetStream`, for the tests whose subject is
|
||||
//! the server's own behaviour: whether it accepts a stream or consumer config,
|
||||
//! and what it delivers.
|
||||
//!
|
||||
//! The same shape as `swarm-controller`'s `test_nats`: the binary comes from
|
||||
//! [`NATS_SERVER_ENV`], which `nix/checks.nix` and `nix/devshell.nix` export,
|
||||
//! and a test that needs it panics when it is unset rather than skipping.
|
||||
|
||||
use std::path::Path;
|
||||
use std::process::{Child, Command, Stdio};
|
||||
use std::time::Duration;
|
||||
|
||||
/// The variable naming the `nats-server` binary.
|
||||
const NATS_SERVER_ENV: &str = "HIVE_NATS_SERVER";
|
||||
|
||||
/// The subdirectory holding nothing but the server's ports file.
|
||||
const PORTS_DIR: &str = "ports";
|
||||
|
||||
/// A running server on a loopback port it bound itself. Dropping it kills
|
||||
/// the server and removes its directory.
|
||||
pub(crate) struct TestNats {
|
||||
child: Child,
|
||||
dir: tempfile::TempDir,
|
||||
}
|
||||
|
||||
impl TestNats {
|
||||
pub(crate) fn start() -> Self {
|
||||
let bin = std::env::var(NATS_SERVER_ENV).unwrap_or_else(|_| {
|
||||
panic!("{NATS_SERVER_ENV} is exported by nix/checks.nix and nix/devshell.nix")
|
||||
});
|
||||
let dir = tempfile::Builder::new()
|
||||
.prefix("test-nats-")
|
||||
.tempdir()
|
||||
.expect("a fresh directory for the server");
|
||||
std::fs::create_dir(dir.path().join(PORTS_DIR)).expect("the ports file directory");
|
||||
let child = Command::new(&bin)
|
||||
.args(["-a", "127.0.0.1", "-p", "-1", "-js", "-sd"])
|
||||
.arg(dir.path().join("store"))
|
||||
.arg("--ports_file_dir")
|
||||
.arg(dir.path().join(PORTS_DIR))
|
||||
.stdout(Stdio::null())
|
||||
.stderr(Stdio::null())
|
||||
.spawn()
|
||||
.unwrap_or_else(|e| panic!("starting {bin}: {e}"));
|
||||
Self { child, dir }
|
||||
}
|
||||
|
||||
/// A client connected to the server, once it accepts connections.
|
||||
pub(crate) async fn client(&self) -> async_nats::Client {
|
||||
for _ in 0..100 {
|
||||
if let Some(url) = announced_url(&self.dir.path().join(PORTS_DIR))
|
||||
&& let Ok(client) = async_nats::connect(&url).await
|
||||
{
|
||||
return client;
|
||||
}
|
||||
tokio::time::sleep(Duration::from_millis(100)).await;
|
||||
}
|
||||
panic!(
|
||||
"nats-server in {} never accepted a connection",
|
||||
self.dir.path().display()
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// The client URL in the ports file the server writes once it listens.
|
||||
fn announced_url(dir: &Path) -> Option<String> {
|
||||
let ports = std::fs::read_dir(dir).ok()?.next()?.ok()?;
|
||||
let ports: serde_json::Value =
|
||||
serde_json::from_slice(&std::fs::read(ports.path()).ok()?).ok()?;
|
||||
ports["nats"][0].as_str().map(str::to_owned)
|
||||
}
|
||||
|
||||
impl Drop for TestNats {
|
||||
fn drop(&mut self) {
|
||||
let _ = self.child.kill();
|
||||
let _ = self.child.wait();
|
||||
}
|
||||
}
|
||||
87
swarm-wire/src/forge_events.rs
Normal file
87
swarm-wire/src/forge_events.rs
Normal file
|
|
@ -0,0 +1,87 @@
|
|||
//! The forge events stream: every verified `issues` and `pull_request`
|
||||
//! delivery swarm-controller receives, kept for the notification router.
|
||||
//!
|
||||
//! The controller checks a delivery's HMAC and publishes its body unchanged
|
||||
//! on [`subject`], one subject per Forgejo event type (the
|
||||
//! `X-Forgejo-Event-Type` header: `issues`, `issue_assign`, `pull_request`,
|
||||
//! `pull_request_sync`, …). The router reads the stream with one durable
|
||||
//! consumer, so it holds no webhook secret and registers no hook.
|
||||
//!
|
||||
//! Three crates name these: the controller publishes, the router consumes,
|
||||
//! and the auth-callout responder grants each its side.
|
||||
|
||||
use std::time::Duration;
|
||||
|
||||
/// The stream the deliveries are kept in.
|
||||
pub const STREAM: &str = "SWARM_EVENTS";
|
||||
|
||||
/// Every subject [`STREAM`] captures sits below this.
|
||||
pub const SUBJECT_PREFIX: &str = "$SWARM.events.forge";
|
||||
|
||||
/// How long the stream keeps a delivery.
|
||||
pub const MAX_AGE: Duration = Duration::from_hours(7 * 24);
|
||||
|
||||
/// The router's durable consumer on [`STREAM`].
|
||||
pub const ROUTER_CONSUMER: &str = "notify-router";
|
||||
|
||||
/// The subject a delivery of Forgejo event type `event_type` is kept under.
|
||||
#[must_use]
|
||||
pub fn subject(event_type: &str) -> String {
|
||||
format!("{SUBJECT_PREFIX}.{event_type}")
|
||||
}
|
||||
|
||||
/// Every [`subject`] at once: the stream's subjects, the router's consumer
|
||||
/// filter and the controller's publish grant.
|
||||
#[must_use]
|
||||
pub fn subject_wildcard() -> String {
|
||||
subject("*")
|
||||
}
|
||||
|
||||
/// The event type a [`subject`] names, or `None` for a subject not below
|
||||
/// [`SUBJECT_PREFIX`].
|
||||
#[must_use]
|
||||
pub fn event_type(subject: &str) -> Option<&str> {
|
||||
subject
|
||||
.strip_prefix(SUBJECT_PREFIX)?
|
||||
.strip_prefix('.')
|
||||
.filter(|t| !t.is_empty() && !t.contains('.'))
|
||||
}
|
||||
|
||||
/// Whether `event_type` can be one token of a subject. The value comes from
|
||||
/// a request header, and a `.`, a wildcard or whitespace would publish
|
||||
/// somewhere other than one leaf below [`SUBJECT_PREFIX`].
|
||||
#[must_use]
|
||||
pub fn is_subject_token(event_type: &str) -> bool {
|
||||
!event_type.is_empty()
|
||||
&& event_type
|
||||
.bytes()
|
||||
.all(|b| b.is_ascii_lowercase() || b.is_ascii_digit() || b == b'_')
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn a_subject_round_trips_to_its_event_type() {
|
||||
assert_eq!(subject("issue_assign"), "$SWARM.events.forge.issue_assign");
|
||||
assert_eq!(subject_wildcard(), "$SWARM.events.forge.*");
|
||||
assert_eq!(
|
||||
event_type("$SWARM.events.forge.issue_assign"),
|
||||
Some("issue_assign")
|
||||
);
|
||||
assert_eq!(event_type("$SWARM.events.forge."), None);
|
||||
assert_eq!(event_type("$SWARM.events.forge.a.b"), None);
|
||||
assert_eq!(event_type("$SWARM.term.atlas"), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn only_a_plain_event_type_is_a_subject_token() {
|
||||
for good in ["issues", "issue_assign", "pull_request_review_approved"] {
|
||||
assert!(is_subject_token(good), "{good}");
|
||||
}
|
||||
for bad in ["", "a.b", "*", ">", "a b", "Issues"] {
|
||||
assert!(!is_subject_token(bad), "{bad:?}");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -16,8 +16,11 @@ pub mod agent_status;
|
|||
pub mod agent_term;
|
||||
pub mod builder;
|
||||
pub mod claude_login;
|
||||
pub mod forge_events;
|
||||
pub mod notices;
|
||||
pub mod notify_router;
|
||||
pub mod review_classifier;
|
||||
pub mod router_todo;
|
||||
pub mod subagent_term;
|
||||
pub mod wanted;
|
||||
|
||||
|
|
|
|||
11
swarm-wire/src/notify_router.rs
Normal file
11
swarm-wire/src/notify_router.rs
Normal file
|
|
@ -0,0 +1,11 @@
|
|||
//! The notification router's own KV bucket, which the auth-callout responder
|
||||
//! grants it alone.
|
||||
//!
|
||||
//! Keyed by issue, it holds the assignees the router last saw on that issue
|
||||
//! and the agents it queued a todo for. Forgejo's assign and unassign
|
||||
//! deliveries carry the assignees after the change and not which one changed,
|
||||
//! so the router works that out from the snapshot; and the closed flow clears
|
||||
//! the todos of the agents listed. It is not a delivery path.
|
||||
|
||||
/// The bucket.
|
||||
pub const STATE_BUCKET: &str = "notify-router-state";
|
||||
112
swarm-wire/src/router_todo.rs
Normal file
112
swarm-wire/src/router_todo.rs
Normal file
|
|
@ -0,0 +1,112 @@
|
|||
//! The router's todo queue: one work-queue stream, one subject and one
|
||||
//! durable consumer per agent, and the op each message carries.
|
||||
//!
|
||||
//! The notification router publishes a [`RouterTodoOp`] on [`subject`] for
|
||||
//! the agent it addresses. That agent's harness drains its own consumer,
|
||||
//! [`consumer_name`], applies each op to its todo store as subsystem
|
||||
//! [`TODO_SUBSYSTEM`], and acks it, which removes it from the stream. An
|
||||
//! agent that is down finds its ops waiting, up to [`MAX_AGE`].
|
||||
//!
|
||||
//! The store dedups on `(subsystem, key)`, so a redelivered op is harmless.
|
||||
//!
|
||||
//! Three crates name these: the router publishes, the harness consumes, and
|
||||
//! the auth-callout responder grants each its side.
|
||||
|
||||
use std::time::Duration;
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
/// The stream the ops are queued in. The router creates it.
|
||||
pub const STREAM: &str = "ROUTER_TODOS";
|
||||
|
||||
/// Every subject [`STREAM`] captures sits below this, one token per agent.
|
||||
pub const SUBJECT_PREFIX: &str = "$SWARM.router.todo";
|
||||
|
||||
/// How long an op waits for its agent to take it.
|
||||
pub const MAX_AGE: Duration = Duration::from_hours(30 * 24);
|
||||
|
||||
/// How many ops one agent's subject holds before the oldest go.
|
||||
pub const MAX_MSGS_PER_SUBJECT: i64 = 1000;
|
||||
|
||||
/// The router's publish dedup window: an op republished with the same
|
||||
/// `Nats-Msg-Id` inside it is dropped by the stream.
|
||||
pub const DUPLICATE_WINDOW: Duration = Duration::from_mins(2);
|
||||
|
||||
/// The todo subsystem every op is applied under, so the router's todos never
|
||||
/// collide with another producer's keys.
|
||||
pub const TODO_SUBSYSTEM: &str = "router";
|
||||
|
||||
/// The subject the router queues `agent`'s ops on.
|
||||
#[must_use]
|
||||
pub fn subject(agent: &str) -> String {
|
||||
format!("{SUBJECT_PREFIX}.{agent}")
|
||||
}
|
||||
|
||||
/// Every [`subject`] at once: the stream's subjects and the router's publish
|
||||
/// grant.
|
||||
#[must_use]
|
||||
pub fn subject_wildcard() -> String {
|
||||
subject("*")
|
||||
}
|
||||
|
||||
/// The durable consumer `agent`'s harness drains: `todo-<agent>`.
|
||||
#[must_use]
|
||||
pub fn consumer_name(agent: &str) -> String {
|
||||
format!("todo-{agent}")
|
||||
}
|
||||
|
||||
/// What an op does to the agent's todo list.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum Op {
|
||||
/// Add the todo, or update the one under the same key.
|
||||
Upsert,
|
||||
/// Remove the todo under the key, or with [`RouterTodoOp::prefix`] every
|
||||
/// router todo whose key starts with it.
|
||||
Clear,
|
||||
}
|
||||
|
||||
/// One change to one agent's router todos.
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct RouterTodoOp {
|
||||
pub op: Op,
|
||||
/// The todo's key under [`TODO_SUBSYSTEM`], such as
|
||||
/// `forge:o/r#7:assignment`.
|
||||
pub key: String,
|
||||
/// For [`Op::Clear`]: `key` is a prefix. Ignored for an upsert.
|
||||
#[serde(default)]
|
||||
pub prefix: bool,
|
||||
/// The line the agent reads. Empty for a clear.
|
||||
#[serde(default)]
|
||||
pub summary: String,
|
||||
/// Which flow produced the op: `router:<flow-id>`.
|
||||
pub source: String,
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn the_names_are_the_agreed_ones() {
|
||||
assert_eq!(subject("atlas"), "$SWARM.router.todo.atlas");
|
||||
assert_eq!(subject_wildcard(), "$SWARM.router.todo.*");
|
||||
assert_eq!(consumer_name("atlas"), "todo-atlas");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_op_reads_back_with_defaults_for_what_a_clear_leaves_out() {
|
||||
let op: RouterTodoOp = serde_json::from_str(
|
||||
r#"{"op":"clear","key":"forge:o/r#1:","prefix":true,"source":"router:closed"}"#,
|
||||
)
|
||||
.expect("decodes");
|
||||
assert_eq!(op.op, Op::Clear);
|
||||
assert!(op.prefix);
|
||||
assert_eq!(op.summary, "");
|
||||
let upsert: RouterTodoOp = serde_json::from_str(
|
||||
r#"{"op":"upsert","key":"k","summary":"s","source":"router:assigned"}"#,
|
||||
)
|
||||
.expect("decodes");
|
||||
assert!(!upsert.prefix);
|
||||
}
|
||||
}
|
||||
Loading…
Reference in a new issue