vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

refactor(sock): one socket client, retry as a policy value

Browse source 
Six places in the tree hand-rolled the same connect / write one JSON
line / read one JSON line back. Two of them — the harness serve loop's
client and the MCP server's — were byte-identical apart from a six-line
wrapper, ~145 lines of literal copy-paste. The other four each
reimplemented a subset, and the subsets had drifted: some named the
socket path in their errors and some did not, one classified transient
against fatal failures and the rest retried nothing at all, two drained
the response and two decoded it.

That duplication was defended when the daemons were split out, on the
grounds that a daemon's socket etiquette should stay visible in the
crate that depends on it. The etiquette genuinely does differ. The code
does not, and five copies is where "each daemon documents its own
etiquette" stops paying for itself.

`hive-sock-client` now owns the transport once, generic over the
request and response types so it is protocol-agnostic: the host-served
control socket and the harness's in-agent socket both use it with their
own wire-type crates. The two real differences become values instead of
forks. Retry is `Retry::RideOutRestart` (2/4/8/16/30s, sized to ride out
a service restart) for callers with no natural retry of their own, or
`Retry::None` for callers already inside a poll loop where the poll
interval is the retry — and the reason each caller picked one is a
comment at the call site rather than a reimplementation. The response is
either decoded (`request`) or half-closed and drained (`notify`, where
the drain exists so the server's write-back doesn't land on a closed
socket). Whether a failure propagates or is logged and swallowed stays
at the call site, because that is the caller's choice and not a property
of the transport.

Errors always name the socket path now, everywhere. That detail is
load-bearing: a permission problem on a socket that reads as "is the
daemon running?" sends the operator to fix the wrong thing.

The transient-against-fatal enum is gone rather than moved. Serialising
happens before the retry loop and deserialising after it, so only
connect, I/O and short-read failures can reach the loop at all — a
deterministic failure is now unretryable by construction instead of by
classification.

It is deliberately a new crate and not part of `hive-agent-sock`. The
`*-sock` crates are pure wire types by convention — `hive-agent-sock`
depends on serde and nothing else — and the two largest copies talk to
the host socket, whose types live in a different crate entirely. A
transport in either wire-type crate would drag tokio into it and point
the wrong way besides.

No wire-format change: same JSON line in, same line out.
This commit is contained in:
atlas 2026-07-26 22:20:01 +02:00 • committed by mara
commit 7a826f9ee2
23 changed files with 498 additions and 501 deletions
Download patch file Download diff file Expand all files Collapse all files

167
hive-agent/src/client.rs
View file

@ -1,167 +0,0 @@
use std::path::Path;
use std::time::Duration;
use anyhow::{Result, anyhow};
use serde::Serialize;
use serde::de::DeserializeOwned;
use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader};
use tokio::net::UnixStream;
/// Backoff schedule between attempts. Five entries → up to 5 retries on
/// top of the initial attempt; total wall-clock cap = 2+4+8+16+30 = 60s.
/// Sized to ride out a hive-c0re restart (systemd usually has the unix
/// socket back inside ~5s) without the agent-side claude session having
/// to handle the transient itself — burning tokens on a tool-error retry
/// loop is more expensive than 60s of in-harness sleep.
const RETRY_BACKOFFS_MS: &[u64] = &[2_000, 4_000, 8_000, 16_000, 30_000];
/// Transparent retry wrapper around [`request_retried`] that throws away
/// the retry count. Use this from non-tool callers (the harness serve
/// loop, web UI, CLI subcommands) where we just want the socket-restart
/// resilience without surfacing the bookkeeping.
///
/// # Errors
///
/// Returns an error if the socket is unreachable after all retries, or if
/// serialization / deserialization of the request or response fails.
pub async fn request<Req, Resp>(socket: &Path, req: &Req) -> Result<Resp>
where
Req: Serialize + ?Sized,
Resp: DeserializeOwned,
{
request_retried(socket, req).await.map(|(resp, _)| resp)
}
/// Same wire shape as [`request`], but reports how many retries it took
/// past the initial attempt (0 = succeeded first try). MCP tool handlers
/// use this so they can append a one-line hint to the tool result when
/// retries happened — that way claude knows the prior socket flake
/// wasn't a content error and shouldn't trigger an LLM-level retry of
/// its own.
///
/// # Errors
///
/// Returns an error if all retries are exhausted, or on a fatal protocol
/// error (serialization / deserialization failure).
///
/// # Panics
///
/// Panics if `RETRY_BACKOFFS_MS.len()` does not fit in a `u32`, which
/// cannot happen with the current compile-time constant.
pub async fn request_retried<Req, Resp>(socket: &Path, req: &Req) -> Result<(Resp, u32)>
where
Req: Serialize + ?Sized,
Resp: DeserializeOwned,
{
let mut last_err: Option<anyhow::Error> = None;
let max_retries = u32::try_from(RETRY_BACKOFFS_MS.len()).unwrap();
for attempt in 0..=max_retries {
match try_once::<Req, Resp>(socket, req).await {
Ok(resp) => return Ok((resp, attempt)),
Err(RequestError::Fatal(e)) => return Err(e),
Err(RequestError::Transient(e)) => {
if attempt < max_retries {
let sleep_ms = RETRY_BACKOFFS_MS[attempt as usize];
tracing::warn!(
attempt = attempt + 1,
sleep_ms,
error = %e,
"hive socket attempt failed; retrying"
);
last_err = Some(e);
tokio::time::sleep(Duration::from_millis(sleep_ms)).await;
} else {
last_err = Some(e);
}
}
}
}
// Reaching here means the final attempt returned `Transient`, which always
// sets `last_err` — so this is infallible.
Err(last_err.expect("a transient failure on the final attempt set last_err"))
}
/// Transient = connect / IO error worth a retry (server restart, broken
/// pipe). Fatal = serialization / deserialization / protocol error
/// where retrying would just repeat the same failure.
enum RequestError {
Transient(anyhow::Error),
Fatal(anyhow::Error),
}
async fn try_once<Req, Resp>(socket: &Path, req: &Req) -> Result<Resp, RequestError>
where
Req: Serialize + ?Sized,
Resp: DeserializeOwned,
{
let stream = match UnixStream::connect(socket).await {
Ok(stream) => stream,
Err(e) => {
// A refused or missing socket usually means hive-c0re is
// mid-restart (operator redeploy / rebuild) — the socket is
// recreated on its boot and `request_retried` rides it out. When
// the error *does* surface (retries exhausted, or a non-retried
// caller) add that context so claude reads it as a likely
// transient rather than a hard failure worth escalating.
let restarting = matches!(
e.kind(),
std::io::ErrorKind::ConnectionRefused | std::io::ErrorKind::NotFound
);
let mut err = anyhow::Error::new(e).context(format!("connect to {}", socket.display()));
if restarting {
err = err.context(
"hive-c0re may be restarting (e.g. an operator redeploy); \
the harness already retried ~60s before surfacing this",
);
}
return Err(RequestError::Transient(err));
}
};
let (read, mut write) = stream.into_split();
let mut payload = serde_json::to_string(req).map_err(|e| RequestError::Fatal(e.into()))?;
payload.push('\n');
write
.write_all(payload.as_bytes())
.await
.map_err(|e| RequestError::Transient(e.into()))?;
write
.flush()
.await
.map_err(|e| RequestError::Transient(e.into()))?;
let mut reader = BufReader::new(read);
let mut line = String::new();
let read_bytes = reader
.read_line(&mut line)
.await
.map_err(|e| RequestError::Transient(e.into()))?;
if read_bytes == 0 || line.is_empty() {
return Err(RequestError::Transient(anyhow!(
"server closed connection without responding"
)));
}
serde_json::from_str(line.trim()).map_err(|e| RequestError::Fatal(e.into()))
}
#[cfg(test)]
mod tests {
use super::{RequestError, try_once};
/// A connect to a non-existent socket path (ENOENT → `NotFound`) is
/// classified transient AND annotated with the "hive-c0re is restarting"
/// hint, so a surfaced tool error reads as the expected transient.
#[tokio::test]
async fn missing_socket_connect_is_transient_with_restart_hint() {
let bogus = std::path::Path::new("/nonexistent/hive/mcp.sock");
match try_once::<(), serde_json::Value>(bogus, &()).await {
Err(RequestError::Transient(e)) => {
let msg = format!("{e:#}");
assert!(msg.contains("restarting"), "missing restart hint: {msg}");
assert!(msg.contains("connect to"), "missing connect context: {msg}");
}
Err(RequestError::Fatal(e)) => panic!("expected transient, got fatal: {e:#}"),
Ok(_) => panic!("expected connect failure to a non-existent socket"),
}
}
}

41
hive-agent/src/main.rs
View file

@ -9,7 +9,6 @@
//! Single bin crate: the module tree below (formerly this crate's `lib.rs`,
//! before lib + bin were collapsed into one) plus the serve loop.
mod client;
mod db_migrate;
mod disk_watch;
mod events;
@ -37,6 +36,12 @@ mod web_ui;
/// Default socket path inside the container — bind-mounted by `hive-c0re`.
const DEFAULT_SOCKET: &str = "/run/hive/mcp.sock";
/// Retry policy for every request to the host-served control socket.
/// Nothing on this side of the socket has a natural retry — the serve loop
/// and the web UI both hand a failure straight to a human or to claude —
/// so a hive-c0re restart is worth waiting out rather than surfacing.
const CONTROL_SOCKET_RETRY: Retry = Retry::RideOutRestart;
/// Default web UI port — used when `HIVE_PORT` env is unset.
const DEFAULT_WEB_PORT: u16 = 8042;
@ -57,6 +62,7 @@ use anyhow::Result;
use clap::Parser;
use hive_core_agent_sock::{Request, Response};
use hive_sh4re::{HelperEvent, SYSTEM_SENDER};
use hive_sock_client::Retry;
#[derive(Parser)]
#[command(name = "hive-agent", about = "hyperhive harness serve loop")]
@ -358,7 +364,7 @@ struct AgentSurface;
/// the `Surface` methods that don't need the reply (`ack_turn`,
/// `requeue_inflight`, `graceful_stop_complete`).
async fn fire_and_forget(socket: &Path, req: Request, label: &str) {
match client::request::<_, Response>(socket, &req).await {
match hive_sock_client::request::<_, Response>(socket, &req, CONTROL_SOCKET_RETRY).await {
Ok(Response::Ok) => {}
Ok(Response::Err { message }) => {
tracing::warn!(%message, "{label} rejected by broker");
@ -387,20 +393,29 @@ impl Surface for AgentSurface {
}
async fn inbox_unread(socket: &Path) -> u64 {
match client::request::<_, Response>(socket, &Request::Status).await {
match hive_sock_client::request::<_, Response>(
socket,
&Request::Status,
CONTROL_SOCKET_RETRY,
)
.await
{
Ok(Response::Status { unread }) => unread,
_ => 0,
}
}
async fn post_turn_counts(socket: &Path) -> (Option<u64>, Option<u64>) {
let threads =
match client::request::<_, Response>(socket, &Request::GetLooseEnds { agent: None })
.await
{
Ok(Response::LooseEnds { loose_ends }) => u64::try_from(loose_ends.len()).ok(),
_ => None,
};
let threads = match hive_sock_client::request::<_, Response>(
socket,
&Request::GetLooseEnds { agent: None },
CONTROL_SOCKET_RETRY,
)
.await
{
Ok(Response::LooseEnds { loose_ends }) => u64::try_from(loose_ends.len()).ok(),
_ => None,
};
// Reminders are harness-local — dial the in-agent socket directly
// instead of the broker.
let reminders =
@ -412,13 +427,14 @@ impl Surface for AgentSurface {
}
async fn send_to_parent(socket: &Path, body: String) {
let res = client::request::<_, Response>(
let res = hive_sock_client::request::<_, Response>(
socket,
&Request::Send {
to: hive_sh4re::PARENT_RECIPIENT.into(),
body,
in_reply_to: None,
},
CONTROL_SOCKET_RETRY,
)
.await;
if let Err(e) = res {
@ -427,12 +443,13 @@ impl Surface for AgentSurface {
}
async fn recv_next(socket: &Path) -> RecvOutcome {
let recv: Result<Response> = client::request(
let recv: Result<Response> = hive_sock_client::request(
socket,
&Request::Recv {
wait_seconds: Some(180),
max: None,
},
CONTROL_SOCKET_RETRY,
)
.await;
match recv {

14
hive-agent/src/todo_server.rs
View file

@ -65,17 +65,13 @@ pub(crate) async fn dial(req: &Request) -> Option<Response> {
if !path.exists() {
return None;
}
tokio::time::timeout(std::time::Duration::from_secs(3), async move {
let mut stream = UnixStream::connect(&path).await.ok()?;
let mut line = serde_json::to_string(req).ok()?;
line.push('\n');
stream.write_all(line.as_bytes()).await.ok()?;
let mut lines = BufReader::new(stream).lines();
let resp_line = lines.next_line().await.ok()??;
serde_json::from_str(&resp_line).ok()
})
tokio::time::timeout(
std::time::Duration::from_secs(3),
hive_sock_client::request::<_, Response>(&path, req, hive_sock_client::Retry::None),
)
.await
.ok()?
.ok()
}
/// Run the in-agent socket server: bind + accept loop, one request/response

6
hive-agent/src/web_ui/mod.rs
View file

@ -305,7 +305,11 @@ async fn broker_request(
) -> std::result::Result<hive_core_agent_sock::Response, BrokerError> {
match tokio::time::timeout(
SOCKET_FETCH_TIMEOUT,
crate::client::request::<_, hive_core_agent_sock::Response>(socket, req),
hive_sock_client::request::<_, hive_core_agent_sock::Response>(
socket,
req,
crate::CONTROL_SOCKET_RETRY,
),
)
.await
{