vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

agents get a todo when assigned to an issue or PR, rewritten when unassigned and cleared on close

Browse source 
A new swarm service, swarm-notify-router, in its own container on the host
that sets deploy.notify-router.enable (defaulted on by allSwarmServices where
the host has the router's store leaf, like the claude login minter).

- swarm-controller registers instance-wide hooks for `issues` +
  `issue_assign` and `pull_request` + `pull_request_assign`, updates the
  event list of a hook already registered with fewer (leaving its active flag as it is), and keeps each
  verified body unchanged in the SWARM_EVENTS stream on
  $SWARM.events.forge.<X-Forgejo-Event-Type>, deduped by X-Forgejo-Delivery.
  A delivery the stream did not keep answers 503.
- swarm-notify-router drains its durable consumer on that stream, runs its
  flows (source -> match -> todo sink, stored as nodes plus edges and checked
  to be a tree) and queues each op on $SWARM.router.todo.<agent> in the
  work-queue stream ROUTER_TODOS. Built-in flows: assigned, unassigned (same
  key, rewritten) and closed (prefix-clears every holder's router todos). A
  per-issue snapshot in its notify-router-state bucket holds the last
  assignees and the todo holders; only agents in the agent-status bucket get
  ops, and an assignee that was not one is left out of the snapshot so the
  next assign event retries it. After the first event that must be retried,
  the rest of the batch is nakked so redelivery keeps stream order.
- hive-agent creates its own durable consumer todo-<agent> and applies each
  op to its todo store as subsystem `router`, then acks; a refused op stops
  the rest of its batch the same way. The store's (subsystem, key) dedup
  makes a redelivery harmless. New Todos::clear_prefix.
- swarm-nats-auth: a --notify-router-client arm; the controller may publish
  $SWARM.events.forge.*; each agent gets three {agent} templates (extended
  consumer create whose filter the server checks, MSG.NEXT, ACK).
- Credentials: a notify-router store leaf minted by glue-bao-tls, a
  swarm-notify-router bao policy (read on its own queue client secret), an
  authelia machine client, the secret publisher entry, a journal forwarder
  route. No manual step.

hive-forge-notify is unchanged, so agents also still get forge's own
notifications.

Refs #5571
This commit is contained in:
atlas 2026-10-22 03:03:26 +02:00 • committed by mara
commit 0f7b403803
53 changed files with 3587 additions and 96 deletions
Download patch file Download diff file Expand all files Collapse all files

211
swarm-controller/src/forge.rs
View file

@ -26,11 +26,11 @@ use forgejo_api::structs::{
AddCollaboratorOption, AddCollaboratorOptionPermission, ChangeFileOperation,
ChangeFileOperationOperation, ChangeFilesOptions, CommitStatusState,
CreateBranchProtectionOption, CreateHookOption, CreateHookOptionConfig, CreateHookOptionType,
CreateRepoOption, CreateStatusOption, CreateUserOption, EditUserOption, GetTreeQuery, GitEntry,
IssueListIssuesQuery, IssueListIssuesQueryState, IssueListIssuesQueryType,
RepoDownloadPullDiffOrPatchQuery, RepoGetContentsQuery, RepoGetPullRequestCommitsQuery,
RepoGetPullRequestFilesQuery, RepoListPullRequestsQuery, RepoListPullRequestsQueryState,
RepoSearchQuery, StateType,
CreateRepoOption, CreateStatusOption, CreateUserOption, EditHookOption, EditUserOption,
GetTreeQuery, GitEntry, IssueListIssuesQuery, IssueListIssuesQueryState,
IssueListIssuesQueryType, RepoDownloadPullDiffOrPatchQuery, RepoGetContentsQuery,
RepoGetPullRequestCommitsQuery, RepoGetPullRequestFilesQuery, RepoListPullRequestsQuery,
RepoListPullRequestsQueryState, RepoSearchQuery, StateType,
};
use forgejo_api::{ApiErrorKind, ForgejoError};
use futures_util::{StreamExt as _, TryStreamExt as _};
@ -1614,29 +1614,8 @@ impl Client {
async fn ensure_all_swarm_hooks(&self, public_base: &str, secret: &str) -> Result<()> {
for kind in DeliveryKind::ALL {
let target_url = kind.target_url(public_base);
let (event, scope) = match kind {
DeliveryKind::Knowledge => (
"push",
HookScope::Repo {
org: KNOWLEDGE_ORG,
repo: KNOWLEDGE_REPO,
},
),
DeliveryKind::ConfigPr => ("pull_request", HookScope::Org { org: CONFIG_ORG }),
// Instance-wide: commit/push activity is not scoped to one
// org (unlike the two above) — it needs observing across
// every repo the forge hosts, per the "no metric exists for
// commits or pushes" tracker issue. Forgejo's
// "global (system) webhook" (`admin_create_hook`) is exactly
// this — the only scope in this API that fires for a repo
// in an org created after this hook was registered, with no
// per-org registration to keep in sync as orgs come and go.
DeliveryKind::VcsActivity => ("push", HookScope::Instance),
// Instance-wide too: any repo whose admin gave the
// classifier's account access can request its review.
DeliveryKind::ReviewRequest => ("pull_request_review_request", HookScope::Instance),
};
self.ensure_hook(&scope, &target_url, event, secret)
let (events, scope) = hook_registration(kind);
self.ensure_hook(&scope, &target_url, events, secret)
.await
.with_context(|| format!("register swarm webhook {target_url}"))?;
}
@ -1650,7 +1629,7 @@ impl Client {
&self,
scope: &HookScope<'_>,
target_url: &str,
event: &str,
events: &[&str],
secret: &str,
) -> Result<()> {
// ⚠️ Both non-matching arms log at a level the journal keeps, because
@ -1667,15 +1646,27 @@ impl Client {
// [`HookScope::extra_create_config`], which is where the instance
// scope's own version of that went wrong), a non-zero count with no
// match means the recorded url is not the one compared.
match scope.list_hook_urls(&self.api).await {
Ok(urls) => {
if urls.iter().any(|u| u == target_url) {
tracing::debug!(%target_url, "swarm forge: webhook already registered");
//
// A listed hook missing one of `events` has its event list replaced,
// so a hook registered before an event was added here gets it.
match scope.list_hooks(&self.api).await {
Ok(hooks) => {
if let Some(hook) = hooks.iter().find(|h| h.url == target_url) {
let missing = missing_events(events, &hook.events);
if missing.is_empty() {
tracing::debug!(%target_url, "swarm forge: webhook already registered");
return Ok(());
}
let id = hook.id.with_context(|| {
format!("the forge listed the hook for {target_url} without an id")
})?;
scope.edit_hook_events(&self.api, id, events).await?;
tracing::info!(%target_url, ?missing, "swarm forge: webhook events updated");
return Ok(());
}
tracing::info!(
%target_url,
listed = urls.len(),
listed = hooks.len(),
"swarm forge: no listed hook matches; registering"
);
}
@ -1704,12 +1695,12 @@ impl Client {
.with_context(|| format!("parse webhook target url {target_url}"))?,
additional,
},
events: Some(vec![event.to_owned()]),
events: Some(events.iter().map(|e| (*e).to_owned()).collect()),
r#type: CreateHookOptionType::Forgejo,
};
match scope.create_hook(&self.api, hook).await {
Ok(()) => {
tracing::info!(%target_url, %event, "swarm forge: webhook registered");
tracing::info!(%target_url, ?events, "swarm forge: webhook registered");
Ok(())
}
Err(e) if is_already_exists(&e) => {
@ -1744,6 +1735,60 @@ fn transitive_reach(start: i64, adj: &HashMap<i64, Vec<i64>>) -> usize {
visited.len() - 1
}
/// The events a swarm hook of `kind` is registered for, and the scope it
/// lives on.
fn hook_registration(kind: DeliveryKind) -> (&'static [&'static str], HookScope<'static>) {
match kind {
DeliveryKind::Knowledge => (
&["push"],
HookScope::Repo {
org: KNOWLEDGE_ORG,
repo: KNOWLEDGE_REPO,
},
),
DeliveryKind::ConfigPr => (&["pull_request"], HookScope::Org { org: CONFIG_ORG }),
// Instance-wide: commit/push activity is not scoped to one
// org (unlike the two above) — it needs observing across
// every repo the forge hosts, per the "no metric exists for
// commits or pushes" tracker issue. Forgejo's
// "global (system) webhook" (`admin_create_hook`) is exactly
// this — the only scope in this API that fires for a repo
// in an org created after this hook was registered, with no
// per-org registration to keep in sync as orgs come and go.
DeliveryKind::VcsActivity => (&["push"], HookScope::Instance),
// Instance-wide too: any repo whose admin gave the
// classifier's account access can request its review.
DeliveryKind::ReviewRequest => (&["pull_request_review_request"], HookScope::Instance),
// Instance-wide, for the notification router, which reads the assign
// events. They are named: Forgejo's web form shows assignment as a
// box of its own, so a hook edited there can drop it while keeping
// `issues` or `pull_request`.
DeliveryKind::Issues => (&["issues", "issue_assign"], HookScope::Instance),
DeliveryKind::PullRequest => (
&["pull_request", "pull_request_assign"],
HookScope::Instance,
),
}
}
/// The events of `wanted` that a hook listed with `listed` lacks. Forgejo
/// lists a hook by its specific event types, so `issue_assign` shows in the
/// list on its own.
fn missing_events<'a>(wanted: &[&'a str], listed: &[String]) -> Vec<&'a str> {
wanted
.iter()
.filter(|w| !listed.iter().any(|l| l == *w))
.copied()
.collect()
}
/// One hook as [`HookScope::list_hooks`] reads it.
struct ListedHook {
id: Option<i64>,
url: String,
events: Vec<String>,
}
/// Where a hook lives. The knowledge hook is repo-scoped and the config-PR
/// hook is org-scoped — a hook on the wrong scope would never fire, and forgejo would
/// report that as a perfectly healthy hook with no deliveries.
@ -1775,7 +1820,7 @@ impl HookScope<'_> {
/// hook — while `GET /admin/hooks` returns only webhooks with the flag
/// set. Omitting it therefore breaks both halves at once: the hook is
/// not the instance-wide one this scope exists for, and
/// [`Self::list_hook_urls`] can never see it, so every process start
/// [`Self::list_hooks`] can never see it, so every process start
/// creates another one — twelve in a day, `listed=0` each time,
/// before this key was sent.
fn extra_create_config(&self) -> &'static [(&'static str, &'static str)] {
@ -1785,8 +1830,8 @@ impl HookScope<'_> {
}
}
/// The `url` config value of every hook currently on this scope.
async fn list_hook_urls(&self, api: &api::Api) -> Result<Vec<String>, ForgejoError> {
/// Every hook currently on this scope that has a `url` config value.
async fn list_hooks(&self, api: &api::Api) -> Result<Vec<ListedHook>, ForgejoError> {
let hooks = match self {
Self::Repo { org, repo } => {
api.call(|api| Box::pin(api.repo_list_hooks(org, repo).all()))
@ -1799,11 +1844,51 @@ impl HookScope<'_> {
Self::Instance => api.call(|api| api.admin_list_hooks().into_future()).await?,
};
Ok(hooks
.iter()
.filter_map(|h| h.config.as_ref()?.get("url").cloned())
.into_iter()
.filter_map(|h| {
Some(ListedHook {
url: h.config.as_ref()?.get("url").cloned()?,
id: h.id,
events: h.events.unwrap_or_default(),
})
})
.collect())
}
/// Replace the event list of the hook `id` on this scope with `events`,
/// leaving its url, secret and content type as they are.
async fn edit_hook_events(
&self,
api: &api::Api,
id: i64,
events: &[&str],
) -> Result<(), ForgejoError> {
let edit = EditHookOption {
active: None,
authorization_header: None,
branch_filter: None,
config: None,
events: Some(events.iter().map(|e| (*e).to_owned()).collect()),
};
match self {
Self::Repo { org, repo } => api
.call(|api| {
api.repo_edit_hook(org, repo, id, edit.clone())
.into_future()
})
.await
.map(drop),
Self::Org { org } => api
.call(|api| api.org_edit_hook(org, id, edit.clone()).into_future())
.await
.map(drop),
Self::Instance => api
.call(|api| api.admin_edit_hook(id, edit.clone()).into_future())
.await
.map(drop),
}
}
async fn create_hook(
&self,
api: &api::Api,
@ -2011,6 +2096,50 @@ impl Client {
mod tests {
use super::*;
/// The event list each swarm hook is registered with, as Forgejo names
/// the event types (`modules/webhook/type.go`).
#[test]
fn each_swarm_hook_is_registered_for_its_named_events() {
let events = |kind| hook_registration(kind).0.to_vec();
assert_eq!(events(DeliveryKind::Knowledge), ["push"]);
assert_eq!(events(DeliveryKind::ConfigPr), ["pull_request"]);
assert_eq!(events(DeliveryKind::VcsActivity), ["push"]);
assert_eq!(
events(DeliveryKind::ReviewRequest),
["pull_request_review_request"]
);
assert_eq!(events(DeliveryKind::Issues), ["issues", "issue_assign"]);
assert_eq!(
events(DeliveryKind::PullRequest),
["pull_request", "pull_request_assign"]
);
}
#[test]
fn a_listed_hook_lacking_a_wanted_event_is_updated_and_one_with_more_is_not() {
let listed = |l: &[&str]| l.iter().map(|s| (*s).to_owned()).collect::<Vec<_>>();
let wanted = ["issues", "issue_assign"];
assert_eq!(
missing_events(&wanted, &listed(&["issues"])),
["issue_assign"]
);
assert_eq!(missing_events(&wanted, &listed(&[])), wanted);
// How Forgejo lists a hook registered as `issues`.
assert!(
missing_events(
&wanted,
&listed(&[
"issues",
"issue_assign",
"issue_label",
"issue_milestone",
"issue_comment"
])
)
.is_empty()
);
}
#[test]
fn open_pulls_are_listed_newest_first_and_skip_one_with_no_number() {
// The generated type insists on `requested_reviewers` being present.