vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

agents get a todo when assigned to an issue or PR, rewritten when unassigned and cleared on close

Browse source 
A new swarm service, swarm-notify-router, in its own container on the host
that sets deploy.notify-router.enable (defaulted on by allSwarmServices where
the host has the router's store leaf, like the claude login minter).

- swarm-controller registers instance-wide hooks for `issues` +
  `issue_assign` and `pull_request` + `pull_request_assign`, updates the
  event list of a hook already registered with fewer (leaving its active flag as it is), and keeps each
  verified body unchanged in the SWARM_EVENTS stream on
  $SWARM.events.forge.<X-Forgejo-Event-Type>, deduped by X-Forgejo-Delivery.
  A delivery the stream did not keep answers 503.
- swarm-notify-router drains its durable consumer on that stream, runs its
  flows (source -> match -> todo sink, stored as nodes plus edges and checked
  to be a tree) and queues each op on $SWARM.router.todo.<agent> in the
  work-queue stream ROUTER_TODOS. Built-in flows: assigned, unassigned (same
  key, rewritten) and closed (prefix-clears every holder's router todos). A
  per-issue snapshot in its notify-router-state bucket holds the last
  assignees and the todo holders; only agents in the agent-status bucket get
  ops, and an assignee that was not one is left out of the snapshot so the
  next assign event retries it. After the first event that must be retried,
  the rest of the batch is nakked so redelivery keeps stream order.
- hive-agent creates its own durable consumer todo-<agent> and applies each
  op to its todo store as subsystem `router`, then acks; a refused op stops
  the rest of its batch the same way. The store's (subsystem, key) dedup
  makes a redelivery harmless. New Todos::clear_prefix.
- swarm-nats-auth: a --notify-router-client arm; the controller may publish
  $SWARM.events.forge.*; each agent gets three {agent} templates (extended
  consumer create whose filter the server checks, MSG.NEXT, ACK).
- Credentials: a notify-router store leaf minted by glue-bao-tls, a
  swarm-notify-router bao policy (read on its own queue client secret), an
  authelia machine client, the secret publisher entry, a journal forwarder
  route. No manual step.

hive-forge-notify is unchanged, so agents also still get forge's own
notifications.

Refs #5571
This commit is contained in:
atlas 2026-10-22 03:03:26 +02:00 • committed by mara
commit 0f7b403803
53 changed files with 3587 additions and 96 deletions
Download patch file Download diff file Expand all files Collapse all files

3
swarm-controller/Cargo.toml
View file

@ -107,11 +107,14 @@ swarm-controller-api = { workspace = true, features = ["schema"] }
# config the crate spells.
# `agent-term`: likewise every agent's terminal stream, plus the consumer it
# replays that stream through.
# `forge-events`: the stream it keeps verified forge deliveries in, for the
# notification router.
swarm-queue-client = { workspace = true, features = [
"kv",
"agent-permissions",
"subagent-term",
"agent-term",
"forge-events",
] }
# `matrix_account.rs` writes the credential this daemon's route accepts. Same
# crate the hive reads it back with, which is the point: the path, the field

211
swarm-controller/src/forge.rs
View file

@ -26,11 +26,11 @@ use forgejo_api::structs::{
AddCollaboratorOption, AddCollaboratorOptionPermission, ChangeFileOperation,
ChangeFileOperationOperation, ChangeFilesOptions, CommitStatusState,
CreateBranchProtectionOption, CreateHookOption, CreateHookOptionConfig, CreateHookOptionType,
CreateRepoOption, CreateStatusOption, CreateUserOption, EditUserOption, GetTreeQuery, GitEntry,
IssueListIssuesQuery, IssueListIssuesQueryState, IssueListIssuesQueryType,
RepoDownloadPullDiffOrPatchQuery, RepoGetContentsQuery, RepoGetPullRequestCommitsQuery,
RepoGetPullRequestFilesQuery, RepoListPullRequestsQuery, RepoListPullRequestsQueryState,
RepoSearchQuery, StateType,
CreateRepoOption, CreateStatusOption, CreateUserOption, EditHookOption, EditUserOption,
GetTreeQuery, GitEntry, IssueListIssuesQuery, IssueListIssuesQueryState,
IssueListIssuesQueryType, RepoDownloadPullDiffOrPatchQuery, RepoGetContentsQuery,
RepoGetPullRequestCommitsQuery, RepoGetPullRequestFilesQuery, RepoListPullRequestsQuery,
RepoListPullRequestsQueryState, RepoSearchQuery, StateType,
};
use forgejo_api::{ApiErrorKind, ForgejoError};
use futures_util::{StreamExt as _, TryStreamExt as _};
@ -1614,29 +1614,8 @@ impl Client {
async fn ensure_all_swarm_hooks(&self, public_base: &str, secret: &str) -> Result<()> {
for kind in DeliveryKind::ALL {
let target_url = kind.target_url(public_base);
let (event, scope) = match kind {
DeliveryKind::Knowledge => (
"push",
HookScope::Repo {
org: KNOWLEDGE_ORG,
repo: KNOWLEDGE_REPO,
},
),
DeliveryKind::ConfigPr => ("pull_request", HookScope::Org { org: CONFIG_ORG }),
// Instance-wide: commit/push activity is not scoped to one
// org (unlike the two above) — it needs observing across
// every repo the forge hosts, per the "no metric exists for
// commits or pushes" tracker issue. Forgejo's
// "global (system) webhook" (`admin_create_hook`) is exactly
// this — the only scope in this API that fires for a repo
// in an org created after this hook was registered, with no
// per-org registration to keep in sync as orgs come and go.
DeliveryKind::VcsActivity => ("push", HookScope::Instance),
// Instance-wide too: any repo whose admin gave the
// classifier's account access can request its review.
DeliveryKind::ReviewRequest => ("pull_request_review_request", HookScope::Instance),
};
self.ensure_hook(&scope, &target_url, event, secret)
let (events, scope) = hook_registration(kind);
self.ensure_hook(&scope, &target_url, events, secret)
.await
.with_context(|| format!("register swarm webhook {target_url}"))?;
}
@ -1650,7 +1629,7 @@ impl Client {
&self,
scope: &HookScope<'_>,
target_url: &str,
event: &str,
events: &[&str],
secret: &str,
) -> Result<()> {
// ⚠️ Both non-matching arms log at a level the journal keeps, because
@ -1667,15 +1646,27 @@ impl Client {
// [`HookScope::extra_create_config`], which is where the instance
// scope's own version of that went wrong), a non-zero count with no
// match means the recorded url is not the one compared.
match scope.list_hook_urls(&self.api).await {
Ok(urls) => {
if urls.iter().any(|u| u == target_url) {
tracing::debug!(%target_url, "swarm forge: webhook already registered");
//
// A listed hook missing one of `events` has its event list replaced,
// so a hook registered before an event was added here gets it.
match scope.list_hooks(&self.api).await {
Ok(hooks) => {
if let Some(hook) = hooks.iter().find(|h| h.url == target_url) {
let missing = missing_events(events, &hook.events);
if missing.is_empty() {
tracing::debug!(%target_url, "swarm forge: webhook already registered");
return Ok(());
}
let id = hook.id.with_context(|| {
format!("the forge listed the hook for {target_url} without an id")
})?;
scope.edit_hook_events(&self.api, id, events).await?;
tracing::info!(%target_url, ?missing, "swarm forge: webhook events updated");
return Ok(());
}
tracing::info!(
%target_url,
listed = urls.len(),
listed = hooks.len(),
"swarm forge: no listed hook matches; registering"
);
}
@ -1704,12 +1695,12 @@ impl Client {
.with_context(|| format!("parse webhook target url {target_url}"))?,
additional,
},
events: Some(vec![event.to_owned()]),
events: Some(events.iter().map(|e| (*e).to_owned()).collect()),
r#type: CreateHookOptionType::Forgejo,
};
match scope.create_hook(&self.api, hook).await {
Ok(()) => {
tracing::info!(%target_url, %event, "swarm forge: webhook registered");
tracing::info!(%target_url, ?events, "swarm forge: webhook registered");
Ok(())
}
Err(e) if is_already_exists(&e) => {
@ -1744,6 +1735,60 @@ fn transitive_reach(start: i64, adj: &HashMap<i64, Vec<i64>>) -> usize {
visited.len() - 1
}
/// The events a swarm hook of `kind` is registered for, and the scope it
/// lives on.
fn hook_registration(kind: DeliveryKind) -> (&'static [&'static str], HookScope<'static>) {
match kind {
DeliveryKind::Knowledge => (
&["push"],
HookScope::Repo {
org: KNOWLEDGE_ORG,
repo: KNOWLEDGE_REPO,
},
),
DeliveryKind::ConfigPr => (&["pull_request"], HookScope::Org { org: CONFIG_ORG }),
// Instance-wide: commit/push activity is not scoped to one
// org (unlike the two above) — it needs observing across
// every repo the forge hosts, per the "no metric exists for
// commits or pushes" tracker issue. Forgejo's
// "global (system) webhook" (`admin_create_hook`) is exactly
// this — the only scope in this API that fires for a repo
// in an org created after this hook was registered, with no
// per-org registration to keep in sync as orgs come and go.
DeliveryKind::VcsActivity => (&["push"], HookScope::Instance),
// Instance-wide too: any repo whose admin gave the
// classifier's account access can request its review.
DeliveryKind::ReviewRequest => (&["pull_request_review_request"], HookScope::Instance),
// Instance-wide, for the notification router, which reads the assign
// events. They are named: Forgejo's web form shows assignment as a
// box of its own, so a hook edited there can drop it while keeping
// `issues` or `pull_request`.
DeliveryKind::Issues => (&["issues", "issue_assign"], HookScope::Instance),
DeliveryKind::PullRequest => (
&["pull_request", "pull_request_assign"],
HookScope::Instance,
),
}
}
/// The events of `wanted` that a hook listed with `listed` lacks. Forgejo
/// lists a hook by its specific event types, so `issue_assign` shows in the
/// list on its own.
fn missing_events<'a>(wanted: &[&'a str], listed: &[String]) -> Vec<&'a str> {
wanted
.iter()
.filter(|w| !listed.iter().any(|l| l == *w))
.copied()
.collect()
}
/// One hook as [`HookScope::list_hooks`] reads it.
struct ListedHook {
id: Option<i64>,
url: String,
events: Vec<String>,
}
/// Where a hook lives. The knowledge hook is repo-scoped and the config-PR
/// hook is org-scoped — a hook on the wrong scope would never fire, and forgejo would
/// report that as a perfectly healthy hook with no deliveries.
@ -1775,7 +1820,7 @@ impl HookScope<'_> {
/// hook — while `GET /admin/hooks` returns only webhooks with the flag
/// set. Omitting it therefore breaks both halves at once: the hook is
/// not the instance-wide one this scope exists for, and
/// [`Self::list_hook_urls`] can never see it, so every process start
/// [`Self::list_hooks`] can never see it, so every process start
/// creates another one — twelve in a day, `listed=0` each time,
/// before this key was sent.
fn extra_create_config(&self) -> &'static [(&'static str, &'static str)] {
@ -1785,8 +1830,8 @@ impl HookScope<'_> {
}
}
/// The `url` config value of every hook currently on this scope.
async fn list_hook_urls(&self, api: &api::Api) -> Result<Vec<String>, ForgejoError> {
/// Every hook currently on this scope that has a `url` config value.
async fn list_hooks(&self, api: &api::Api) -> Result<Vec<ListedHook>, ForgejoError> {
let hooks = match self {
Self::Repo { org, repo } => {
api.call(|api| Box::pin(api.repo_list_hooks(org, repo).all()))
@ -1799,11 +1844,51 @@ impl HookScope<'_> {
Self::Instance => api.call(|api| api.admin_list_hooks().into_future()).await?,
};
Ok(hooks
.iter()
.filter_map(|h| h.config.as_ref()?.get("url").cloned())
.into_iter()
.filter_map(|h| {
Some(ListedHook {
url: h.config.as_ref()?.get("url").cloned()?,
id: h.id,
events: h.events.unwrap_or_default(),
})
})
.collect())
}
/// Replace the event list of the hook `id` on this scope with `events`,
/// leaving its url, secret and content type as they are.
async fn edit_hook_events(
&self,
api: &api::Api,
id: i64,
events: &[&str],
) -> Result<(), ForgejoError> {
let edit = EditHookOption {
active: None,
authorization_header: None,
branch_filter: None,
config: None,
events: Some(events.iter().map(|e| (*e).to_owned()).collect()),
};
match self {
Self::Repo { org, repo } => api
.call(|api| {
api.repo_edit_hook(org, repo, id, edit.clone())
.into_future()
})
.await
.map(drop),
Self::Org { org } => api
.call(|api| api.org_edit_hook(org, id, edit.clone()).into_future())
.await
.map(drop),
Self::Instance => api
.call(|api| api.admin_edit_hook(id, edit.clone()).into_future())
.await
.map(drop),
}
}
async fn create_hook(
&self,
api: &api::Api,
@ -2011,6 +2096,50 @@ impl Client {
mod tests {
use super::*;
/// The event list each swarm hook is registered with, as Forgejo names
/// the event types (`modules/webhook/type.go`).
#[test]
fn each_swarm_hook_is_registered_for_its_named_events() {
let events = |kind| hook_registration(kind).0.to_vec();
assert_eq!(events(DeliveryKind::Knowledge), ["push"]);
assert_eq!(events(DeliveryKind::ConfigPr), ["pull_request"]);
assert_eq!(events(DeliveryKind::VcsActivity), ["push"]);
assert_eq!(
events(DeliveryKind::ReviewRequest),
["pull_request_review_request"]
);
assert_eq!(events(DeliveryKind::Issues), ["issues", "issue_assign"]);
assert_eq!(
events(DeliveryKind::PullRequest),
["pull_request", "pull_request_assign"]
);
}
#[test]
fn a_listed_hook_lacking_a_wanted_event_is_updated_and_one_with_more_is_not() {
let listed = |l: &[&str]| l.iter().map(|s| (*s).to_owned()).collect::<Vec<_>>();
let wanted = ["issues", "issue_assign"];
assert_eq!(
missing_events(&wanted, &listed(&["issues"])),
["issue_assign"]
);
assert_eq!(missing_events(&wanted, &listed(&[])), wanted);
// How Forgejo lists a hook registered as `issues`.
assert!(
missing_events(
&wanted,
&listed(&[
"issues",
"issue_assign",
"issue_label",
"issue_milestone",
"issue_comment"
])
)
.is_empty()
);
}
#[test]
fn open_pulls_are_listed_newest_first_and_skip_one_with_no_number() {
// The generated type insists on `requested_reviewers` being present.

120
swarm-controller/src/forge_events.rs Normal file
View file

@ -0,0 +1,120 @@
//! Keeping verified `issues` and `pull_request` deliveries for the
//! notification router.
//!
//! [`crate::webhook`] checks the HMAC, then [`publish`] puts the body,
//! unchanged, on `swarm_wire::forge_events::subject(<event type>)` in the
//! `SWARM_EVENTS` stream. The router reads that stream, so it needs no
//! webhook secret and registers no hook of its own.
//!
//! The event type is Forgejo's `X-Forgejo-Event-Type` header (`issue_assign`,
//! `pull_request_sync`, …), not `X-Forgejo-Event`, which folds those into
//! `issues` and `pull_request`. It reaches the subject only once it is one
//! plain token.
use axum::http::HeaderMap;
use std::time::Duration;
/// The header naming the delivery's event type.
const EVENT_TYPE_HEADER: &str = "x-forgejo-event-type";
/// The header naming the delivery. A redelivery carries the same one, and the
/// stream drops a second message with the same id.
const DELIVERY_HEADER: &str = "x-forgejo-delivery";
/// How long the stream may take to ack a publish. Inside Forgejo's own
/// delivery timeout.
const PUBLISH_DEADLINE: Duration = Duration::from_secs(5);
/// Why a delivery was not kept.
#[derive(Debug, PartialEq, Eq)]
pub(crate) enum NotKept {
/// No `X-Forgejo-Event-Type`, or one that is not a single subject token.
BadEventType,
/// The controller has no queue, or the stream refused or timed out.
Queue(String),
}
/// The subject a delivery with `headers` goes to.
fn subject(headers: &HeaderMap) -> Result<String, NotKept> {
let event_type = headers
.get(EVENT_TYPE_HEADER)
.and_then(|v| v.to_str().ok())
.filter(|t| swarm_wire::forge_events::is_subject_token(t))
.ok_or(NotKept::BadEventType)?;
Ok(swarm_wire::forge_events::subject(event_type))
}
/// Keep one verified delivery in the stream, and wait for the stream's ack.
pub(crate) async fn publish(
state: &crate::routes::AppState,
headers: &HeaderMap,
body: axum::body::Bytes,
) -> Result<(), NotKept> {
let subject = subject(headers)?;
let Some(status) = state.status.as_ref() else {
return Err(NotKept::Queue("no swarm queue is configured".to_owned()));
};
let client = status.queue_client();
swarm_queue_client::ensure_connected(&client).map_err(|e| NotKept::Queue(e.to_string()))?;
let mut message = async_nats::jetstream::message::PublishMessage::build().payload(body);
if let Some(id) = headers.get(DELIVERY_HEADER).and_then(|v| v.to_str().ok()) {
message = message.message_id(id);
}
let kept = async {
swarm_queue_client::forge_events::open_or_create(&client)
.await
.map_err(|e| swarm_queue_client::chain(&e))?;
async_nats::jetstream::new(client.clone())
.send_publish(subject.clone(), message)
.await
.map_err(|e| swarm_queue_client::chain(&e))?
.await
.map_err(|e| swarm_queue_client::chain(&e))
};
match tokio::time::timeout(PUBLISH_DEADLINE, kept).await {
Ok(Ok(ack)) => {
tracing::info!(%subject, seq = ack.sequence, duplicate = ack.duplicate, "webhook: forge event kept");
Ok(())
}
Ok(Err(e)) => Err(NotKept::Queue(e)),
Err(_) => Err(NotKept::Queue(format!(
"the stream did not ack within {}s",
PUBLISH_DEADLINE.as_secs()
))),
}
}
#[cfg(test)]
mod tests {
use super::{NotKept, subject};
use axum::http::{HeaderMap, HeaderValue};
fn with_type(value: &str) -> HeaderMap {
let mut headers = HeaderMap::new();
headers.insert(
"X-Forgejo-Event-Type",
HeaderValue::from_str(value).expect("a header value"),
);
headers
}
#[test]
fn the_event_type_header_names_the_subject() {
assert_eq!(
subject(&with_type("issue_assign")),
Ok("$SWARM.events.forge.issue_assign".to_owned())
);
}
#[test]
fn a_missing_or_multi_token_event_type_is_not_kept() {
assert_eq!(subject(&HeaderMap::new()), Err(NotKept::BadEventType));
for bad in ["issues.x", ">", "*", ""] {
assert_eq!(
subject(&with_type(bad)),
Err(NotKept::BadEventType),
"{bad:?}"
);
}
}
}

1
swarm-controller/src/main.rs
View file

@ -42,6 +42,7 @@ mod claude_login;
mod config_pr;
mod forge;
mod forge_account;
mod forge_events;
mod github_account;
mod inference;
mod issue_report;

38
swarm-controller/src/webhook.rs
View file

@ -10,6 +10,10 @@
//! the payload and emits a semantic message — *the knowledge repo changed*,
//! *deploy agent X at rev Y* — addressed to the hives that need it. One place
//! decides what a forge payload means, so no hive re-derives it.
//!
//! The `issues` and `pull_request` hooks are the exception: their bodies are
//! kept unchanged in the forge events stream (`crate::forge_events`), and the
//! notification router decides what they mean.
use anyhow::{Context as _, Result};
use axum::{
@ -225,6 +229,14 @@ pub(super) enum DeliveryKind {
/// for the review classifier's account (`crate::review_classifier::review`).
#[strum(serialize = "review-request")]
ReviewRequest,
/// Every `issues` event on every repo in the instance (opened, closed,
/// assigned, …), kept for the notification router (`crate::forge_events`).
#[strum(serialize = "issues")]
Issues,
/// Every `pull_request` event on every repo in the instance, kept the
/// same way.
#[strum(serialize = "pull-request")]
PullRequest,
}
/// The route prefix a registered `target_url` must point at.
@ -240,11 +252,13 @@ impl DeliveryKind {
/// Every kind, for the registration sweep. An array rather than a
/// hand-written list at the call site, so adding a kind cannot leave one
/// hook unregistered.
pub(super) const ALL: [Self; 4] = [
pub(super) const ALL: [Self; 6] = [
Self::Knowledge,
Self::ConfigPr,
Self::VcsActivity,
Self::ReviewRequest,
Self::Issues,
Self::PullRequest,
];
/// The `target_url` to register with Forgejo for this kind, given the
@ -352,18 +366,22 @@ pub(super) fn verify(
/// comment for why both this and a periodic poll write the same cache), and a
/// merge queues a deploy of the merged commit on the agent's hive. A
/// `ReviewRequest` delivery naming the review classifier's account starts a
/// review run in the background.
/// review run in the background. An `Issues` or `PullRequest` delivery is
/// kept in the forge events stream for the notification router
/// (`crate::forge_events`).
///
/// Returns 200 on an accepted delivery so Forgejo does not retry. A refused
/// one answers 401 (bad signature) or 503 (this daemon has no secret), and
/// an unknown `{kind}` answers 404 rather than a cheerful 200.
/// an unknown `{kind}` answers 404 rather than a cheerful 200. An `Issues` or
/// `PullRequest` delivery the stream did not keep answers 503, so the forge's
/// delivery log shows it failed and a redelivery can keep it.
#[utoipa::path(
post,
path = "/webhook/forge/{kind}",
params(
("kind" = String, Path, description = "which swarm-wide hook this delivery arrived on: \
`knowledge`, `config-pr`, `vcs-activity` or \
`review-request`")
`knowledge`, `config-pr`, `vcs-activity`, \
`review-request`, `issues` or `pull-request`")
),
request_body(
content = String,
@ -376,7 +394,9 @@ pub(super) fn verify(
(status = 200, description = "delivery accepted", body = String),
(status = 401, description = "bad or missing HMAC signature"),
(status = 404, description = "unknown hook kind in the path"),
(status = 503, description = "HMAC secret unavailable at startup"),
(status = 503, description = "HMAC secret unavailable at startup, or an `issues` or \
`pull-request` delivery the forge events stream did not \
keep"),
),
tag = "webhook"
)]
@ -430,6 +450,12 @@ pub(super) async fn post_webhook_forge(
}
}
DeliveryKind::ReviewRequest => crate::review_classifier::review::on_delivery(&state, &body),
DeliveryKind::Issues | DeliveryKind::PullRequest => {
if let Err(not_kept) = crate::forge_events::publish(&state, &headers, body).await {
tracing::warn!(kind = <&str>::from(kind), reason = ?not_kept, "webhook: forge event not kept");
return (StatusCode::SERVICE_UNAVAILABLE, "forge event not kept").into_response();
}
}
}
(StatusCode::OK, "ok").into_response()