vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(#2642): a github.com notification poller alongside the forge one

Browse source 
hive-forge-notify grows a second binary, hive-github-notify. The two
share the notification half of the job — tolerant parse, classification,
formatting, dedupe, todo delivery — and nothing else: each binary owns
its host's protocol outright.

Two binaries rather than one multi-source daemon, and rather than a
cargo feature. A feature would unify across the workspace and cost every
crate its build cache. Two binaries keep the decision in nix: forge.nix
installs the forge unit, github.nix installs the github one under
hyperhive.github.enable, so a hive built without that module has no
github poller in its closure at all — GitHub access is separable (a
tier, a policy boundary), not merely switched off. Both binaries ship
from the existing derivation, so packages.nix is untouched.

The split is real at the code level too, not just at the unit level.
source.rs is a trait; the impls live in the binaries that use them, so
neither binary links the other's protocol code and the library names no
host at all. The forge-only assigned-issue rollup moves into the forge
binary for the same reason: it asks the forge what is assigned to this
agent, which is not a notification-protocol concern.

At runtime the github unit needs a PAT at <state>/github-token, the same
dashboard-provisioned token the gh wrapper and the git credential helper
already use. No PAT: it logs why and exits 0, which is why the unit is
Restart=on-failure and not always.

Forgejo's notifications API is modelled on GitHub's, so one tolerant
parse serves both — the differences (string thread ids, PullRequest vs
Pull) are absorbed by lenient deserializers rather than a second parse
path. Thread ids normalise to String at the parse boundary; they are
only ever opaque keys. Todo keys gain a per-source prefix so the two
hosts cannot collide, and the forge's is deliberately empty to keep
existing forge todo keys stable across the deploy that lands this.

The github loop honours the server's X-Poll-Interval, re-arming only
when the server asks for a slower cadence than ours; the hint is read
before the status check, because it arrives on error and empty pages too
and that is exactly when it matters. Reading the notification stream
needs the notifications scope on the PAT, which a token minted for push
access typically lacks; the failure mode is silence, so docs/github.md
says so explicitly.
This commit is contained in:
atlas 2026-07-31 12:05:02 +02:00 • committed by mara
commit 0db83c40a0
15 changed files with 971 additions and 412 deletions
Download patch file Download diff file Expand all files Collapse all files

206
hive-forge-notify/src/bin/hive-forge-notify/main.rs Normal file
View file

@ -0,0 +1,206 @@
//! Notification poller for the hive's internal Forgejo.
//!
//! Polls unread threads, turns each into a todo on the harness's in-agent
//! socket, and marks it read. The shared half — classification, wake
//! formatting, dedupe, delivery — lives in the library; this binary owns
//! the Forgejo protocol (`source.rs`) and the forge-only assigned-issue
//! rollup below.
mod source;
use std::collections::HashMap;
use std::path::Path;
use std::time::Duration;
use hive_forge_notify::notify::{
POLL_INTERVAL_SECS, TOKEN_RETRY_MAX, TOKEN_RETRY_SECS, poll_once, resolve_own_login,
};
use hive_forge_notify::{HTTP_TIMEOUT_SECS, TODO_SOCKET_RETRY, agent_socket, init_tracing};
use source::ForgejoSource;
use tracing::{debug, info, warn};
#[tokio::main]
async fn main() {
init_tracing();
forgejo_loop(hive_forge_notify::state_dir(), agent_socket()).await;
}
/// Returns — ending the process — when the forge is not configured for
/// this agent, which is a supported state and not a failure. The unit is
/// `Restart = on-failure` for exactly this reason.
async fn forgejo_loop(state_dir: String, socket: std::path::PathBuf) {
let forge_url = match std::env::var("HIVE_FORGE_URL") {
Ok(u) if !u.is_empty() => u,
_ => {
debug!("forge_notify: HIVE_FORGE_URL not set — disabled");
return;
}
};
let token_path = format!("{state_dir}/forge-token");
// Retry reading the token to handle races where hive-priv provisions
// it after the harness starts, or where a parent-container chown
// briefly makes the file unreadable.
let token = {
let mut attempts = 0u32;
loop {
match tokio::fs::read_to_string(&token_path).await {
Ok(t) => {
let t = t.trim().to_owned();
if !t.is_empty() {
break t;
}
debug!("forge_notify: empty forge token at {token_path}");
}
Err(e) => debug!("forge_notify: cannot read token at {token_path}: {e}"),
}
attempts += 1;
if attempts >= TOKEN_RETRY_MAX {
debug!(
"forge_notify: token not available after {TOKEN_RETRY_MAX} retries — disabled"
);
return;
}
tokio::time::sleep(Duration::from_secs(TOKEN_RETRY_SECS)).await;
}
};
let Some(source) = ForgejoSource::new(&forge_url, &token) else {
return;
};
let client = match reqwest::Client::builder()
.timeout(Duration::from_secs(HTTP_TIMEOUT_SECS))
.build()
{
Ok(c) => c,
Err(e) => {
warn!("forge_notify: failed to build HTTP client: {e}");
return;
}
};
// Self-notification filtering degrades safely: an empty login means
// no filtering, and a boot-time failure (forge not yet reachable) is
// re-attempted on each tick rather than staying off for the process
// lifetime.
let mut own_login = resolve_own_login(&client, &source).await;
if own_login.is_empty() {
warn!(
"forge_notify: could not resolve own login yet — self-notification \
filtering disabled until it resolves on a later poll"
);
} else {
debug!(%own_login, "forge_notify: own login resolved");
}
let mut interval = tokio::time::interval(Duration::from_secs(POLL_INTERVAL_SECS));
interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay);
// First tick fires immediately — skip it so we don't race the broker
// socket becoming available right at boot.
interval.tick().await;
info!(forge_url = %forge_url, "forge_notify: polling started");
// In-process delivery dedupe: thread id -> the `updated_at` we last
// woke the agent for. Deliberately ephemeral: forge's own read-state
// is the durable record of what's been delivered, so a rebuild starts
// empty and re-scans only the genuinely-still-unread set, which is
// tiny by construction because delivery marks read.
let mut delivered: HashMap<String, String> = HashMap::new();
loop {
interval.tick().await;
if own_login.is_empty() {
own_login = resolve_own_login(&client, &source).await;
if !own_login.is_empty() {
debug!(%own_login, "forge_notify: own login resolved on retry");
}
}
// Forgejo states no poll-interval hint, so the return is ignored
// and the configured cadence stands.
let _ = poll_once(&source, &client, &socket, &mut delivered, &own_login).await;
update_assigned_rollup(&client, &forge_url, &token, &socket).await;
}
}
/// Count of open issues or PRs assigned to this agent, via Forgejo's
/// global search API. `issue_type` is `"issues"` or `"pulls"`. Reads the
/// `X-Total-Count` header rather than deserialising the body — only page 1
/// with limit 1 is fetched, keeping the request cheap. `None` on any error.
async fn count_assigned(
client: &reqwest::Client,
forge_url: &str,
token: &str,
issue_type: &str,
) -> Option<u64> {
let url = format!(
"{forge_url}/api/v1/issues/search\
?type={issue_type}&state=open&assigned=true&limit=1&page=1"
);
let resp = match client
.get(&url)
.header("Authorization", format!("token {token}"))
.send()
.await
{
Ok(r) if r.status().is_success() => r,
_ => return None,
};
let count_str = resp.headers().get("x-total-count")?.to_str().ok()?;
count_str.parse::<u64>().ok()
}
/// Keep a keyed `"rollup"` todo in sync with the count of open assigned
/// issues + PRs: a positive count summarises the breakdown, zero clears
/// the todo. The rollup key is distinct from per-thread keys so clearing
/// it never touches notification todos.
///
/// Forge-only by nature — it asks the forge what is assigned to this
/// agent, which is not a notification-protocol concern and has no
/// github.com counterpart here.
async fn update_assigned_rollup(
client: &reqwest::Client,
forge_url: &str,
token: &str,
socket: &Path,
) {
let issues = count_assigned(client, forge_url, token, "issues")
.await
.unwrap_or(0);
let pulls = count_assigned(client, forge_url, token, "pulls")
.await
.unwrap_or(0);
let total = issues + pulls;
let req = if total == 0 {
hive_agent_sock::Request::ClearTodo {
subsystem: "forge".to_owned(),
key: Some("rollup".to_owned()),
all: false,
}
} else {
let breakdown = match (issues, pulls) {
(i, 0) => format!("{i} issue{}", if i == 1 { "" } else { "s" }),
(0, p) => format!("{p} PR{}", if p == 1 { "" } else { "s" }),
(i, p) => format!(
"{i} issue{}, {p} PR{}",
if i == 1 { "" } else { "s" },
if p == 1 { "" } else { "s" }
),
};
hive_agent_sock::Request::UpsertTodo {
subsystem: "forge".to_owned(),
key: Some("rollup".to_owned()),
summary: format!("{total} open assigned: {breakdown}"),
source: None,
}
};
match hive_sock_client::request::<_, hive_agent_sock::Response>(socket, &req, TODO_SOCKET_RETRY)
.await
{
Ok(_) => debug!(total, "forge_notify: assigned rollup todo updated"),
Err(e) => debug!("forge_notify: assigned rollup todo update failed: {e}"),
}
}

151
hive-forge-notify/src/bin/hive-forge-notify/source.rs Normal file
View file

@ -0,0 +1,151 @@
//! The internal Forgejo's half of the poller — the protocol code that
//! only this binary links.
//!
//! Uses the typed `forgejo-api` client for the calls with stable shapes
//! (identity probe, notification list, mark-read). Pages come back as an
//! opaque `String` and are parsed per-item upstream: one unrepresentable
//! field (a merged PR's `subject.state = "merged"`) must not poison the
//! whole page.
use std::time::Duration;
use forgejo_api::structs::{NotifyGetListQuery, NotifyReadThreadQuery};
use forgejo_api::{Auth, Forgejo, ForgejoError};
use hive_forge_notify::notify::{HTTP_TIMEOUT_SECS, UNREAD_FETCH_LIMIT};
use hive_forge_notify::source::Source;
use tracing::{debug, warn};
/// The hive's internal Forgejo.
pub struct ForgejoSource {
forge: Box<Forgejo>,
/// Base URL, kept for the assigned-issues rollup query, which goes
/// out over plain `reqwest`.
pub base_url: String,
pub token: String,
}
impl ForgejoSource {
/// Build from the internal forge's URL + token. `None` when the URL
/// or client is unusable — the caller treats that as "not
/// configured", not as fatal.
pub fn new(base_url: &str, token: &str) -> Option<Self> {
let parsed = match url::Url::parse(base_url) {
Ok(u) => u,
Err(e) => {
warn!("forge_notify: invalid HIVE_FORGE_URL {base_url}: {e}");
return None;
}
};
match Forgejo::new(Auth::Token(token), parsed) {
Ok(forge) => Some(Self {
forge: Box::new(forge),
base_url: base_url.to_owned(),
token: token.to_owned(),
}),
Err(e) => {
warn!("forge_notify: failed to build forge client: {e}");
None
}
}
}
}
impl Source for ForgejoSource {
/// Deliberately empty: forge todo keys stay the bare thread ids they
/// have always been, so a deploy cannot orphan in-flight todos.
fn key_prefix(&self) -> &'static str {
""
}
fn name(&self) -> &'static str {
"forgejo"
}
fn authorize(&self, rb: reqwest::RequestBuilder) -> reqwest::RequestBuilder {
rb.header("Authorization", format!("token {}", self.token))
}
async fn own_login(&self, client: &reqwest::Client) -> String {
let url = format!("{}/api/v1/user", self.base_url.trim_end_matches('/'));
let Ok(resp) = self.authorize(client.get(&url)).send().await else {
return String::new();
};
if !resp.status().is_success() {
return String::new();
}
resp.json::<serde_json::Value>()
.await
.ok()
.and_then(|v| v["login"].as_str().map(str::to_owned))
.unwrap_or_default()
}
async fn list_unread(
&self,
_client: &reqwest::Client,
) -> Option<(Vec<serde_json::Value>, Option<u64>)> {
let query = NotifyGetListQuery {
all: Some(false),
..NotifyGetListQuery::default()
};
let request = self
.forge
.notify_get_list(query)
.page_size(u32::try_from(UNREAD_FETCH_LIMIT).unwrap_or(u32::MAX))
.response_type::<String>();
let raw = match tokio::time::timeout(Duration::from_secs(HTTP_TIMEOUT_SECS), request.send())
.await
{
Ok(Ok(raw)) => raw,
Ok(Err(ForgejoError::UnexpectedStatusCode(status))) => {
debug!("forge_notify: poll status {status}");
return None;
}
Ok(Err(e)) => {
debug!("forge_notify: poll request failed: {e}");
return None;
}
Err(_) => {
debug!("forge_notify: poll request failed: timed out");
return None;
}
};
match serde_json::from_str(&raw) {
// Forgejo states no poll-interval hint, so the caller keeps
// its own cadence.
Ok(values) => Some((values, None)),
Err(e) => {
warn!("forge_notify: response parse error: {e}");
None
}
}
}
async fn mark_read(&self, _client: &reqwest::Client, id: &str) {
let Ok(thread_id) = id.parse::<i64>() else {
warn!(%id, "forge_notify: non-numeric forgejo thread id — cannot mark read");
return;
};
// `to_status: None` → Forgejo's default unread → read transition.
// The 205 body is the thread JSON, which can carry
// `subject.state = "merged"`; take it as an opaque `String` so a
// merged PR can't turn a successful mark-read into a parse error.
let request = self
.forge
.notify_read_thread(thread_id, NotifyReadThreadQuery { to_status: None })
.response_type::<String>();
match tokio::time::timeout(Duration::from_secs(HTTP_TIMEOUT_SECS), request.send()).await {
Err(_) => {
warn!(%id, "forge_notify: mark-read request failed — notification will resurface");
}
Ok(Err(e @ ForgejoError::ReqwestError(_))) => {
warn!(%id, error = ?e, "forge_notify: mark-read request failed — notification will resurface");
}
Ok(Err(e)) => {
warn!(%id, error = %e, "forge_notify: mark-read returned non-2xx — notification will resurface");
}
Ok(Ok(_)) => debug!(%id, "forge_notify: marked read"),
}
}
}

110
hive-forge-notify/src/bin/hive-github-notify/main.rs Normal file
View file

@ -0,0 +1,110 @@
//! `hive-github-notify` binary — long-running per-agent **github.com**
//! notification poller. Same delivery path as its Forgejo sibling: unread
//! list, per-thread summary, todo upsert on the harness's in-agent socket,
//! mark read on the source.
//!
//! Takes no arguments. `HYPERHIVE_STATE_DIR` holds the PAT
//! (`github-token`, provisioned from the dashboard credentials tab — see
//! `docs/github.md`) and `HIVE_AGENT_SOCKET` is the harness's todo socket.
//! Having a PAT *is* the opt-in: with no token the poller logs why and
//! exits 0, so deploying this unit to an agent that never gets one costs a
//! settled process rather than a restart loop.
//!
//! ⚠️ Reading the notification stream needs the **`notifications` scope**
//! on the PAT — a token minted for `gh` + `git push` usually carries `repo`
//! only, which is enough to push and open PRs but not to read (or mark
//! read) notifications. A PAT without it is not fatal: the poller logs the
//! refusal and stays quiet, so the symptom is silence rather than an error.
mod source;
use std::collections::HashMap;
use std::time::Duration;
use hive_forge_notify::notify::{
POLL_INTERVAL_SECS, TOKEN_RETRY_MAX, TOKEN_RETRY_SECS, poll_once, resolve_own_login,
};
use hive_forge_notify::{HTTP_TIMEOUT_SECS, agent_socket, init_tracing};
use source::GithubSource;
use tracing::{debug, info, warn};
#[tokio::main]
async fn main() {
init_tracing();
let socket = agent_socket();
info!(socket = %socket.display(), "hive-github-notify starting");
// Returns only when no PAT ever arrives; otherwise loops forever.
github_loop(hive_forge_notify::state_dir(), socket).await;
}
/// Waits for the PAT to appear: the token is written out of band from the
/// dashboard and takes effect without a rebuild, so an agent that gains a
/// PAT mid-session starts getting notifications on the next tick rather
/// than after a restart. Gives up — returning, so the process exits 0
/// rather than restart-looping — when no PAT ever arrives, which is the
/// common case for an agent that has the unit but no account.
async fn github_loop(state_dir: String, socket: std::path::PathBuf) {
let token_path = format!("{state_dir}/github-token");
let mut attempts = 0u32;
let token = loop {
match tokio::fs::read_to_string(&token_path).await {
Ok(t) if !t.trim().is_empty() => break t.trim().to_owned(),
_ => debug!("forge_notify: no github token at {token_path} yet"),
}
attempts += 1;
if attempts >= TOKEN_RETRY_MAX {
debug!(
"forge_notify: no github token after {TOKEN_RETRY_MAX} retries — github disabled"
);
return;
}
tokio::time::sleep(Duration::from_secs(TOKEN_RETRY_SECS)).await;
};
let client = match reqwest::Client::builder()
.timeout(Duration::from_secs(HTTP_TIMEOUT_SECS))
.build()
{
Ok(c) => c,
Err(e) => {
warn!("forge_notify: failed to build github HTTP client: {e}");
return;
}
};
let source = GithubSource::new(&token);
let mut own_login = resolve_own_login(&client, &source).await;
let mut delivered: HashMap<String, String> = HashMap::new();
// GitHub tells callers how often it is willing to be polled
// (`X-Poll-Interval`, 60s in practice) and rate-limits those who
// ignore it. Start at our own cadence and re-arm to whatever the
// server asks for — never faster than it wants, never slower than we
// need.
let mut cadence = POLL_INTERVAL_SECS;
let mut interval = tokio::time::interval(Duration::from_secs(cadence));
interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay);
interval.tick().await;
info!("forge_notify: github polling started");
loop {
interval.tick().await;
if own_login.is_empty() {
own_login = resolve_own_login(&client, &source).await;
}
let hint = poll_once(&source, &client, &socket, &mut delivered, &own_login).await;
if let Some(secs) = hint.filter(|s| *s > cadence) {
debug!(
secs,
"forge_notify: github asked for a slower poll — re-arming"
);
cadence = secs;
interval = tokio::time::interval(Duration::from_secs(cadence));
interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay);
interval.tick().await;
}
}
}

132
hive-forge-notify/src/bin/hive-github-notify/source.rs Normal file
View file

@ -0,0 +1,132 @@
//! github.com's half of the poller — the protocol code that only this
//! binary links.
//!
//! Plain `reqwest`: the typed Forgejo client cannot address a different
//! API surface, and GitHub's payloads differ enough (string thread ids,
//! its own `subject.type` vocabulary, no `subject.state`) that the
//! tolerant parse in the shared library is the right common layer, not
//! the client.
use hive_forge_notify::notify::UNREAD_FETCH_LIMIT;
use hive_forge_notify::source::Source;
use tracing::{debug, warn};
/// GitHub's REST base. `docs/github.md` scopes the integration to
/// github.com only — the same constraint the `gh` wrapper and the git
/// credential helper already carry — so this is a constant rather than
/// another operator-entered URL.
const API_BASE: &str = "https://api.github.com";
/// Value of the `X-GitHub-Api-Version` header. GitHub dates its REST
/// versions; pinning one means a future default bump cannot silently
/// reshape the payloads this daemon parses.
const API_VERSION: &str = "2022-11-28";
/// `User-Agent` for GitHub calls. GitHub rejects requests without one.
const USER_AGENT: &str = "hyperhive-forge-notify";
/// github.com, via a personal access token.
pub struct GithubSource {
token: String,
}
impl GithubSource {
pub fn new(token: &str) -> Self {
Self {
token: token.to_owned(),
}
}
}
impl Source for GithubSource {
/// Namespaced, so a github thread id cannot collide with a forge one
/// on a shared todo key.
fn key_prefix(&self) -> &'static str {
"gh:"
}
fn name(&self) -> &'static str {
"github"
}
/// `Bearer`, not Forgejo's `token` — plus the version and user-agent
/// headers GitHub requires. Sending the wrong scheme does not error:
/// it authenticates as *nobody* and silently drops to the
/// unauthenticated rate limit, which is why the rate-limit header is
/// the only cheap way to tell the two apart.
fn authorize(&self, rb: reqwest::RequestBuilder) -> reqwest::RequestBuilder {
rb.header("Authorization", format!("Bearer {}", self.token))
.header("Accept", "application/vnd.github+json")
.header("X-GitHub-Api-Version", API_VERSION)
.header("User-Agent", USER_AGENT)
}
async fn own_login(&self, client: &reqwest::Client) -> String {
let url = format!("{API_BASE}/user");
let Ok(resp) = self.authorize(client.get(&url)).send().await else {
return String::new();
};
if !resp.status().is_success() {
return String::new();
}
resp.json::<serde_json::Value>()
.await
.ok()
.and_then(|v| v["login"].as_str().map(str::to_owned))
.unwrap_or_default()
}
async fn list_unread(
&self,
client: &reqwest::Client,
) -> Option<(Vec<serde_json::Value>, Option<u64>)> {
let url = format!("{API_BASE}/notifications?all=false&per_page={UNREAD_FETCH_LIMIT}");
let resp = match self.authorize(client.get(&url)).send().await {
Ok(resp) => resp,
Err(e) => {
debug!("forge_notify: github poll request failed: {e}");
return None;
}
};
// Read the cadence hint before the status check: GitHub sends it
// on an empty page too, and that is exactly the tick where we
// most want to learn we are polling too fast.
let poll_interval = resp
.headers()
.get("x-poll-interval")
.and_then(|v| v.to_str().ok())
.and_then(|v| v.parse::<u64>().ok());
let status = resp.status();
if !status.is_success() {
// 403 with a rate-limit body and 401 on a revoked PAT are
// both "stay quiet and retry", not fatal: the unit must not
// restart-loop on a credential the operator fixes out of band.
debug!("forge_notify: github poll status {status}");
return None;
}
let raw = resp.text().await.ok()?;
match serde_json::from_str(&raw) {
Ok(values) => Some((values, poll_interval)),
Err(e) => {
warn!("forge_notify: github response parse error: {e}");
None
}
}
}
async fn mark_read(&self, client: &reqwest::Client, id: &str) {
let url = format!("{API_BASE}/notifications/threads/{id}");
match self.authorize(client.patch(&url)).send().await {
Ok(resp) if resp.status().is_success() => {
debug!(%id, "forge_notify: marked read (github)");
}
Ok(resp) => {
warn!(%id, status = %resp.status(), "forge_notify: github mark-read non-2xx — notification will resurface");
}
Err(e) => {
warn!(%id, error = ?e, "forge_notify: github mark-read failed — notification will resurface");
}
}
}
}