agents get a todo when assigned to an issue or PR, rewritten when unassigned and cleared on close

A new swarm service, swarm-notify-router, in its own container on the host
that sets deploy.notify-router.enable (defaulted on by allSwarmServices where
the host has the router's store leaf, like the claude login minter).

- swarm-controller registers instance-wide hooks for `issues` +
  `issue_assign` and `pull_request` + `pull_request_assign`, updates the
  event list of a hook already registered with fewer (leaving its active flag as it is), and keeps each
  verified body unchanged in the SWARM_EVENTS stream on
  $SWARM.events.forge.<X-Forgejo-Event-Type>, deduped by X-Forgejo-Delivery.
  A delivery the stream did not keep answers 503.
- swarm-notify-router drains its durable consumer on that stream, runs its
  flows (source -> match -> todo sink, stored as nodes plus edges and checked
  to be a tree) and queues each op on $SWARM.router.todo.<agent> in the
  work-queue stream ROUTER_TODOS. Built-in flows: assigned, unassigned (same
  key, rewritten) and closed (prefix-clears every holder's router todos). A
  per-issue snapshot in its notify-router-state bucket holds the last
  assignees and the todo holders; only agents in the agent-status bucket get
  ops, and an assignee that was not one is left out of the snapshot so the
  next assign event retries it. After the first event that must be retried,
  the rest of the batch is nakked so redelivery keeps stream order.
- hive-agent creates its own durable consumer todo-<agent> and applies each
  op to its todo store as subsystem `router`, then acks; a refused op stops
  the rest of its batch the same way. The store's (subsystem, key) dedup
  makes a redelivery harmless. New Todos::clear_prefix.
- swarm-nats-auth: a --notify-router-client arm; the controller may publish
  $SWARM.events.forge.*; each agent gets three {agent} templates (extended
  consumer create whose filter the server checks, MSG.NEXT, ACK).
- Credentials: a notify-router store leaf minted by glue-bao-tls, a
  swarm-notify-router bao policy (read on its own queue client secret), an
  authelia machine client, the secret publisher entry, a journal forwarder
  route. No manual step.

hive-forge-notify is unchanged, so agents also still get forge's own
notifications.

Refs #5571

swarm-queue-client/src/forge_events.rs

@@ -0,0 +1,77 @@
+//! Opening the forge events stream. The name, subjects and why it exists are
+//! `swarm_wire::forge_events`.
+
+use swarm_wire::forge_events::{MAX_AGE, STREAM, subject_wildcard};
+
+/// The config [`open_or_create`] creates the stream with.
+fn stream_config() -> async_nats::jetstream::stream::Config {
+    async_nats::jetstream::stream::Config {
+        name: STREAM.to_owned(),
+        description: Some("Verified forge webhook deliveries".to_owned()),
+        subjects: vec![subject_wildcard()],
+        max_age: MAX_AGE,
+        // A redelivered webhook carries the same `X-Forgejo-Delivery`, which
+        // the controller sends as the message id.
+        duplicate_window: std::time::Duration::from_hours(1),
+        ..Default::default()
+    }
+}
+
+/// Open the stream, creating it if nothing has yet. Called by the controller,
+/// which publishes into it, and by the router, which reads it: either may come
+/// up first.
+pub async fn open_or_create(
+    client: &async_nats::Client,
+) -> Result<async_nats::jetstream::stream::Stream, crate::Error> {
+    let js = async_nats::jetstream::new(client.clone());
+    match js.get_stream(STREAM).await {
+        Ok(stream) => Ok(stream),
+        Err(e) => {
+            tracing::info!(stream = STREAM, reason = %e, "forge events stream not available, creating it");
+            js.create_stream(stream_config())
+                .await
+                .map_err(|source| crate::Error::CreateStream {
+                    stream: STREAM,
+                    source,
+                })
+        }
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use swarm_wire::forge_events::{STREAM, subject};
+
+    use super::open_or_create;
+    use crate::test_nats::TestNats;
+
+    /// Either end may create the stream first, and a redelivered webhook is
+    /// kept once.
+    #[tokio::test]
+    async fn the_stream_is_created_once_and_keeps_a_redelivery_once() {
+        let nats = TestNats::start();
+        let client = nats.client().await;
+        open_or_create(&client).await.expect("created");
+        open_or_create(&client).await.expect("opened as it is");
+        let js = async_nats::jetstream::new(client);
+        for (event, id) in [
+            ("issues", "d-1"),
+            ("issues", "d-1"),
+            ("issue_assign", "d-2"),
+        ] {
+            js.send_publish(
+                subject(event),
+                async_nats::jetstream::message::PublishMessage::build()
+                    .payload("{}".into())
+                    .message_id(id),
+            )
+            .await
+            .expect("published")
+            .await
+            .expect("acked");
+        }
+        let mut stream = js.get_stream(STREAM).await.expect("the stream");
+        let info = stream.info().await.expect("info");
+        assert_eq!(info.state.messages, 2);
+    }
+}

swarm-queue-client/src/lib.rs

@@ -130,7 +130,7 @@ pub enum Error {
         source: async_nats::jetstream::kv::WatchError,
     },
 
-    #[cfg(feature = "notices")]
+    #[cfg(any(feature = "notices", feature = "forge-events", feature = "router-todo"))]
     #[error("creating the {stream} stream")]
     CreateStream {
         stream: &'static str,
@@ -464,6 +464,20 @@ pub mod agent_term;
 #[cfg(feature = "subagent-term")]
 pub mod subagent_term;
 
+/// Opening the stream swarm-controller keeps verified forge deliveries in.
+/// The names are `swarm_wire::forge_events`. Behind the `forge-events` feature.
+#[cfg(feature = "forge-events")]
+pub mod forge_events;
+
+/// The notification router's todo queue: the stream, and each agent's own
+/// consumer. The names are `swarm_wire::router_todo`. Behind the
+/// `router-todo` feature.
+#[cfg(feature = "router-todo")]
+pub mod router_todo;
+
+#[cfg(all(test, any(feature = "forge-events", feature = "router-todo")))]
+mod test_nats;
+
 /// Only the fields this needs; authelia returns several.
 #[derive(serde::Deserialize)]
 struct TokenResponse {

swarm-queue-client/src/router_todo.rs

@@ -0,0 +1,203 @@
+//! The router's todo queue from both ends: the router creates the stream, and
+//! each agent's harness creates and drains its own consumer. The names,
+//! subjects and op are `swarm_wire::router_todo`.
+
+use std::time::Duration;
+
+use swarm_wire::router_todo::{
+    DUPLICATE_WINDOW, MAX_AGE, MAX_MSGS_PER_SUBJECT, STREAM, consumer_name, subject,
+    subject_wildcard,
+};
+
+/// How long a delivered op may stay unacked before the server delivers it
+/// again.
+const ACK_WAIT: Duration = Duration::from_secs(30);
+
+/// The config [`open_or_create`] creates the stream with.
+///
+/// Work-queue retention: an op leaves the stream once its agent acks it. The
+/// server then requires every consumer's filter to be disjoint from every
+/// other's, which one consumer per agent subject is.
+fn stream_config() -> async_nats::jetstream::stream::Config {
+    use async_nats::jetstream::stream::{Config, DiscardPolicy, RetentionPolicy};
+
+    Config {
+        name: STREAM.to_owned(),
+        description: Some("Notification router todo ops, one subject per agent".to_owned()),
+        subjects: vec![subject_wildcard()],
+        retention: RetentionPolicy::WorkQueue,
+        max_age: MAX_AGE,
+        max_messages_per_subject: MAX_MSGS_PER_SUBJECT,
+        discard: DiscardPolicy::Old,
+        duplicate_window: DUPLICATE_WINDOW,
+        ..Default::default()
+    }
+}
+
+/// Open the stream, creating it if it does not exist yet. The router is its
+/// one writer and its one creator.
+pub async fn open_or_create(
+    client: &async_nats::Client,
+) -> Result<async_nats::jetstream::stream::Stream, crate::Error> {
+    let js = async_nats::jetstream::new(client.clone());
+    match js.get_stream(STREAM).await {
+        Ok(stream) => Ok(stream),
+        Err(e) => {
+            tracing::info!(stream = STREAM, reason = %e, "router todo stream not available, creating it");
+            js.create_stream(stream_config())
+                .await
+                .map_err(|source| crate::Error::CreateStream {
+                    stream: STREAM,
+                    source,
+                })
+        }
+    }
+}
+
+/// The durable pull consumer `agent`'s harness drains: its own subject, from
+/// the oldest op still queued, each acked explicitly.
+fn consumer_config(agent: &str) -> async_nats::jetstream::consumer::pull::Config {
+    use async_nats::jetstream::consumer::{AckPolicy, DeliverPolicy, pull::Config};
+
+    Config {
+        durable_name: Some(consumer_name(agent)),
+        description: Some(format!("Router todo ops for {agent}")),
+        filter_subject: subject(agent),
+        deliver_policy: DeliverPolicy::All,
+        ack_policy: AckPolicy::Explicit,
+        ack_wait: ACK_WAIT,
+        ..Default::default()
+    }
+}
+
+/// Create `agent`'s consumer, or return it as it is when it already exists
+/// with the same config.
+///
+/// Created on the stream by name, without looking the stream up first: the
+/// agent's grant has no `STREAM.INFO`. The create subject is the extended
+/// form ending in the consumer's filter, the one subject the grant names.
+pub async fn open_own(
+    client: &async_nats::Client,
+    agent: &str,
+) -> Result<
+    async_nats::jetstream::consumer::Consumer<async_nats::jetstream::consumer::pull::Config>,
+    async_nats::jetstream::stream::ConsumerError,
+> {
+    async_nats::jetstream::new(client.clone())
+        .create_consumer_on_stream(consumer_config(agent), STREAM)
+        .await
+}
+
+#[cfg(test)]
+mod tests {
+    use futures_util::StreamExt as _;
+    use swarm_wire::router_todo::{STREAM, consumer_name, subject};
+
+    use super::{consumer_config, open_or_create, open_own};
+    use crate::test_nats::TestNats;
+
+    async fn queue(js: &async_nats::jetstream::Context, agent: &str, id: &str) {
+        js.send_publish(
+            subject(agent),
+            async_nats::jetstream::message::PublishMessage::build()
+                .payload(id.as_bytes().to_vec().into())
+                .message_id(id),
+        )
+        .await
+        .expect("published")
+        .await
+        .expect("acked by the stream");
+    }
+
+    async fn queued(js: &async_nats::jetstream::Context) -> u64 {
+        js.get_stream(STREAM)
+            .await
+            .expect("the stream")
+            .info()
+            .await
+            .expect("its info")
+            .state
+            .messages
+    }
+
+    /// The server takes the stream and both agents' consumers, delivers each
+    /// agent its own ops only, drops a republished op by its id, and removes an
+    /// op once its agent acks it.
+    #[tokio::test]
+    async fn an_agents_consumer_drains_its_own_ops_and_an_ack_removes_them() {
+        let nats = TestNats::start();
+        let client = nats.client().await;
+        let js = async_nats::jetstream::new(client.clone());
+        open_or_create(&client).await.expect("created");
+        open_or_create(&client).await.expect("opened as it is");
+
+        queue(&js, "atlas", "1:assigned:sink:atlas").await;
+        queue(&js, "atlas", "1:assigned:sink:atlas").await;
+        queue(&js, "argus", "1:assigned:sink:argus").await;
+        assert_eq!(queued(&js).await, 2, "the republished op is dropped");
+
+        let atlas = open_own(&client, "atlas").await.expect("atlas's consumer");
+        open_own(&client, "atlas")
+            .await
+            .expect("created again, unchanged");
+        open_own(&client, "argus")
+            .await
+            .expect("argus's, disjoint from atlas's");
+
+        let mut batch = atlas
+            .fetch()
+            .max_messages(10)
+            .messages()
+            .await
+            .expect("a batch");
+        let mut got = Vec::new();
+        while let Some(message) = batch.next().await {
+            let message = message.expect("a message");
+            got.push(message.subject.to_string());
+            message.double_ack().await.expect("acked");
+        }
+        assert_eq!(got, [subject("atlas")]);
+        assert_eq!(queued(&js).await, 1, "argus's op waits for argus");
+    }
+
+    /// The grant's premise: in the extended create form the server refuses a
+    /// payload filter that is not the subject's, so atlas's create subject
+    /// cannot make a consumer that delivers argus's ops.
+    #[tokio::test]
+    async fn the_server_refuses_a_payload_filter_other_than_the_create_subjects() {
+        let nats = TestNats::start();
+        let client = nats.client().await;
+        open_or_create(&client).await.expect("created");
+        let create = |filter: String| {
+            let mut config = consumer_config("atlas");
+            config.filter_subject = filter;
+            serde_json::to_vec(&serde_json::json!({
+                "stream_name": STREAM,
+                "config": async_nats::jetstream::consumer::IntoConsumerConfig::into_consumer_config(config),
+                "action": "",
+            }))
+            .expect("encodes")
+        };
+        let subject_for_atlas = format!(
+            "$JS.API.CONSUMER.CREATE.{STREAM}.{}.{}",
+            consumer_name("atlas"),
+            subject("atlas")
+        );
+        let answer = |body: Vec<u8>| {
+            let client = client.clone();
+            let subject_for_atlas = subject_for_atlas.clone();
+            async move {
+                let reply = client
+                    .request(subject_for_atlas, body.into())
+                    .await
+                    .expect("an answer");
+                serde_json::from_slice::<serde_json::Value>(&reply.payload).expect("json")
+            }
+        };
+        let refused = answer(create(subject("argus"))).await;
+        assert!(refused.get("error").is_some(), "{refused}");
+        // Control: the matching filter is accepted on the same subject.
+        let accepted = answer(create(subject("atlas"))).await;
+        assert!(accepted.get("error").is_none(), "{accepted}");
+    }
+}

swarm-queue-client/src/test_nats.rs

@@ -0,0 +1,78 @@
+//! A throwaway `nats-server` with `JetStream`, for the tests whose subject is
+//! the server's own behaviour: whether it accepts a stream or consumer config,
+//! and what it delivers.
+//!
+//! The same shape as `swarm-controller`'s `test_nats`: the binary comes from
+//! [`NATS_SERVER_ENV`], which `nix/checks.nix` and `nix/devshell.nix` export,
+//! and a test that needs it panics when it is unset rather than skipping.
+
+use std::path::Path;
+use std::process::{Child, Command, Stdio};
+use std::time::Duration;
+
+/// The variable naming the `nats-server` binary.
+const NATS_SERVER_ENV: &str = "HIVE_NATS_SERVER";
+
+/// The subdirectory holding nothing but the server's ports file.
+const PORTS_DIR: &str = "ports";
+
+/// A running server on a loopback port it bound itself. Dropping it kills
+/// the server and removes its directory.
+pub(crate) struct TestNats {
+    child: Child,
+    dir: tempfile::TempDir,
+}
+
+impl TestNats {
+    pub(crate) fn start() -> Self {
+        let bin = std::env::var(NATS_SERVER_ENV).unwrap_or_else(|_| {
+            panic!("{NATS_SERVER_ENV} is exported by nix/checks.nix and nix/devshell.nix")
+        });
+        let dir = tempfile::Builder::new()
+            .prefix("test-nats-")
+            .tempdir()
+            .expect("a fresh directory for the server");
+        std::fs::create_dir(dir.path().join(PORTS_DIR)).expect("the ports file directory");
+        let child = Command::new(&bin)
+            .args(["-a", "127.0.0.1", "-p", "-1", "-js", "-sd"])
+            .arg(dir.path().join("store"))
+            .arg("--ports_file_dir")
+            .arg(dir.path().join(PORTS_DIR))
+            .stdout(Stdio::null())
+            .stderr(Stdio::null())
+            .spawn()
+            .unwrap_or_else(|e| panic!("starting {bin}: {e}"));
+        Self { child, dir }
+    }
+
+    /// A client connected to the server, once it accepts connections.
+    pub(crate) async fn client(&self) -> async_nats::Client {
+        for _ in 0..100 {
+            if let Some(url) = announced_url(&self.dir.path().join(PORTS_DIR))
+                && let Ok(client) = async_nats::connect(&url).await
+            {
+                return client;
+            }
+            tokio::time::sleep(Duration::from_millis(100)).await;
+        }
+        panic!(
+            "nats-server in {} never accepted a connection",
+            self.dir.path().display()
+        );
+    }
+}
+
+/// The client URL in the ports file the server writes once it listens.
+fn announced_url(dir: &Path) -> Option<String> {
+    let ports = std::fs::read_dir(dir).ok()?.next()?.ok()?;
+    let ports: serde_json::Value =
+        serde_json::from_slice(&std::fs::read(ports.path()).ok()?).ok()?;
+    ports["nats"][0].as_str().map(str::to_owned)
+}
+
+impl Drop for TestNats {
+    fn drop(&mut self) {
+        let _ = self.child.kill();
+        let _ = self.child.wait();
+    }
+}