vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

agents: remove send(operator) and the operator inbox; alert on turn failures

Browse source 
Agents reach the operator over matrix now (#5115). A `send` to
`operator` is refused, by the broker and by the MCP allow-list check,
with an error that names the matrix MCP tools. The dashboard's Y3R C4LL
tab goes with it: the operator inbox, `GET /api/operator-inbox`,
`Broker::unread_for_recipient`, and the browser notifications that only
ever fired on operator-bound messages.

The harness no longer posts turn failures and plugin-install failures
to the operator inbox. Both are already logged at warn; a new Grafana
rule, `hh-agent-turn-failed`, fires per agent on the harness's
`turn failed` line.

Dashboard-to-agent messaging (op-send, the agent web UI's message box,
mark-all-read for agent inboxes) is unchanged.

Decisions per mara on #5115: q1 a, q2 a plus a grafana alert, q3 keep
dashboard-to-agent messaging and leave swarm-level code alone.
This commit is contained in:
atlas 2026-10-12 00:44:11 +02:00 • committed by mara
commit 7e6a4c7727
37 changed files with 246 additions and 848 deletions
Download patch file Download diff file Expand all files Collapse all files

33
hive-agent/src/identity.rs
View file

@ -48,20 +48,9 @@ pub fn swarm_name() -> Option<String> {
non_empty_env("HYPERHIVE_SWARM_NAME")
}
/// Hive-qualified agent identity. When the hive domain is configured, returns
/// `${label}@${domain}` (e.g. `iris@darkest.space`); when not, returns just
/// the short label so callers can render a single string regardless of
/// deployment shape. Callers that want to know whether the result is
/// qualified should check [`hive_domain`] directly.
#[must_use]
pub fn qualified_label() -> String {
qualify(&label())
}
/// Apply hive qualification to an arbitrary agent label (e.g. a peer name
/// from the broker). Mirrors [`qualified_label`] but lets a caller qualify
/// names it didn't read from the env. Returns `${label}@${domain}` when the
/// hive domain is set, else just `label`.
/// Apply hive qualification to an agent label (e.g. a peer name from the
/// broker). Returns `${label}@${domain}` when the hive domain is set, else
/// just `label`.
#[must_use]
pub fn qualify(label: &str) -> String {
match hive_domain() {
@ -151,27 +140,27 @@ mod tests {
}
#[test]
fn qualified_label_with_domain_set() {
fn qualify_with_domain_set() {
with_env(Some("iris"), Some("darkest.space"), || {
assert_eq!(qualified_label(), "iris@darkest.space");
assert_eq!(qualify(&label()), "iris@darkest.space");
assert_eq!(hive_domain().as_deref(), Some("darkest.space"));
});
}
#[test]
fn qualified_label_falls_back_to_short_when_domain_unset() {
fn qualify_falls_back_to_short_when_domain_unset() {
with_env(Some("iris"), None, || {
assert_eq!(qualified_label(), "iris");
assert_eq!(qualify(&label()), "iris");
assert!(hive_domain().is_none());
});
}
#[test]
fn qualified_label_falls_back_to_short_when_domain_empty() {
fn qualify_falls_back_to_short_when_domain_empty() {
// Empty string is treated the same as unset — a misconfigured
// module shouldn't surface `iris@` (no domain) to the operator.
with_env(Some("iris"), Some(""), || {
assert_eq!(qualified_label(), "iris");
assert_eq!(qualify(&label()), "iris");
assert!(hive_domain().is_none());
});
}
@ -247,8 +236,8 @@ mod tests {
assert!(hive_domain().is_none());
assert_eq!(hive_name().as_deref(), Some("pr1ma"));
assert_eq!(swarm_name().as_deref(), Some("constellat1on"));
// qualified_label still degrades to short label without domain.
assert_eq!(qualified_label(), "iris");
// qualify still degrades to the short label without a domain.
assert_eq!(qualify(&label()), "iris");
},
);
}

62
hive-agent/src/main.rs
View file

@ -137,22 +137,6 @@ fn log_system_event(bus: &Bus, from: &str, body: &str) {
});
}
/// Body string for the turn-failure notification we route to the
/// operator on `TurnError::Failed`. Reads the hive-qualified
/// identity so the receiver sees `agent@hive` rather than relying on
/// the caller threading a `label` through every turn-handling layer.
/// Falls back to `<unknown>` when `HIVE_LABEL` is missing so a
/// misconfigured harness still produces a parseable line.
fn format_turn_failure(err: &anyhow::Error) -> String {
let who = crate::identity::qualified_label();
let who = if who.is_empty() {
"<unknown>".to_owned()
} else {
who
};
format!("[system] `{who}` claude turn failed:\n{err:#}")
}
/// What a finished turn tells the serve loop to do next.
struct TurnControl {
/// The turn ended in `AuthFailed` — caller parks on login.
@ -293,16 +277,6 @@ trait Surface {
/// fallback. Same shape as `graceful_stop_complete`.
fn pause_acknowledged(socket: &Path) -> impl Future<Output = ()>;
/// Send a message addressed to the operator. The reporting line out
/// of a container: this is where a turn failure or a plugin-install
/// failure surfaces when nothing inside the harness can act on it.
///
/// Was `<parent>` while the hierarchy existed, a sentinel the broker resolved per
/// `topology.json` and which already fell through to `operator` for a
/// root agent. With the parent field gone every agent takes that
/// branch, so the recipient is written out rather than resolved.
fn send_to_operator(socket: &Path, body: String) -> impl Future<Output = ()>;
/// Long-poll the broker for the next message. Wraps the
/// `Messages`/empty/error trichotomy in `RecvOutcome` so the
/// generic `serve_loop` doesn't need the per-role Response enum
@ -387,22 +361,6 @@ impl Surface for AgentSurface {
(threads, reminders)
}
async fn send_to_operator(socket: &Path, body: String) {
let res = hive_sock_client::request::<_, Response>(
socket,
&Request::Send {
to: hive_sh4re::manager::OPERATOR_RECIPIENT.into(),
body,
in_reply_to: None,
},
CONTROL_SOCKET_RETRY,
)
.await;
if let Err(e) = res {
tracing::warn!(error = ?e, "failed to notify the operator of turn failure");
}
}
async fn recv_next(socket: &Path) -> RecvOutcome {
let recv: Result<Response> = hive_sock_client::request_within(
socket,
@ -548,11 +506,7 @@ async fn serve_main<S: Surface>(socket: &Path, poll_ms: u64) -> Result<()> {
}
}
let files = turn::TurnFiles::prepare(socket, &label).await?;
// Plugin install failures come back as a Vec<String> — route each
// through the `send_to_operator` failure-notify path.
for failure in plugins::install_configured().await {
S::send_to_operator(socket, failure).await;
}
plugins::install_configured().await;
// The forge notification poller used to be spawned here. It is its own
// process now (`hive-forge-notify`, its own systemd unit) so a harness
// restart doesn't take forge notifications down with it; it reaches the
@ -944,7 +898,8 @@ fn apply_todo_wake_checked(checked: Option<bool>, todo_miss_streak: &mut u32, bu
}
/// Drive a single turn: emit boot-of-turn events, run claude, ack on
/// success / requeue on rate-limit-or-401 / notify parent on failure,
/// success / requeue on rate-limit-or-401 / log `turn failed` on a hard
/// failure (the line the `hh-agent-turn-failed` Grafana rule alerts on),
/// record stats. Returns a `TurnControl` carrying the auth-failed flag and
/// the todo-wake-checked signal — the serve loop decides what to do next.
async fn handle_turn<S: Surface>(
@ -1061,9 +1016,11 @@ async fn handle_turn<S: Surface>(
/// The non-happy-path half of `handle_turn`: react to each `TurnError`
/// variant the turn could have failed with (park-and-retry on rate-limit/
/// stall/auth, requeue-for-a-fresh-turn on prompt-too-long/session-not-
/// found, notify the operator on a hard failure). Split out purely to keep
/// `handle_turn` itself under clippy's line-count lint — no behavior
/// change from when this lived inline.
/// found). A hard failure needs nothing here: `turn::emit_turn_end` has
/// already logged `turn failed`, which the `hh-agent-turn-failed` Grafana
/// rule alerts on. Split out purely to keep `handle_turn` itself under
/// clippy's line-count lint — no behavior change from when this lived
/// inline.
async fn handle_turn_error_recovery<S: Surface>(
outcome: &turn::TurnOutcome,
bus: &Bus,
@ -1123,7 +1080,4 @@ async fn handle_turn_error_recovery<S: Surface>(
tracing::warn!("session-not-found; requeueing message for a fresh turn");
S::requeue_inflight(socket).await;
}
if let Err(turn::TurnError::Failed(e)) = outcome {
S::send_to_operator(socket, format_turn_failure(e)).await;
}
}

22
hive-agent/src/plugins.rs
View file

@ -96,23 +96,20 @@ async fn update_marketplaces() {
}
/// Install every plugin in `/etc/hyperhive/claude-plugins.json`.
/// Returns a list of human-readable failure messages so the caller can
/// route them through their own per-role surface (turn-failure-style
/// notification, see `Surface::send_to_operator`). Wire-agnostic: the
/// caller picks the recipient, the same way failure-notify does.
pub async fn install_configured() -> Vec<String> {
/// A failed install is logged and skipped; the harness starts without it.
pub async fn install_configured() {
let Ok(raw) = tokio::fs::read_to_string(PLUGINS_PATH).await else {
return Vec::new();
return;
};
let specs: Vec<String> = match serde_json::from_str(&raw) {
Ok(v) => v,
Err(e) => {
tracing::warn!(path = PLUGINS_PATH, error = ?e, "claude-plugins spec parse failed; skipping");
return Vec::new();
return;
}
};
if specs.is_empty() {
return Vec::new();
return;
}
add_marketplaces().await;
if auto_update_enabled().await {
@ -120,7 +117,6 @@ pub async fn install_configured() -> Vec<String> {
} else {
tracing::debug!("claudePluginsAutoUpdate=false, skipping marketplace update");
}
let mut failures = Vec::new();
for spec in specs {
match Command::new("claude")
.args(["plugin", "install", &spec])
@ -138,18 +134,10 @@ pub async fn install_configured() -> Vec<String> {
stderr = %stderr,
"claude plugin install failed",
);
failures.push(format!(
"claude plugin install failed for `{spec}`:\n{}",
stderr.trim()
));
}
Err(e) => {
tracing::warn!(spec = %spec, error = ?e, "claude plugin install spawn failed");
failures.push(format!(
"claude plugin install spawn failed for `{spec}`: {e}"
));
}
}
}
failures
}

6
hive-agent/src/turn.rs
View file

@ -201,8 +201,8 @@ pub enum TurnError {
/// provider (e.g. HTTP 429) without reporting it ends up here. Handled
/// like [`Self::ApiStall`].
AgentStall(String),
/// A hard failure with no recovery — the serve loop escalates it to the
/// operator (`send_to_operator`).
/// A hard failure with no recovery. Logged as `turn failed`, the exact
/// message the `hh-agent-turn-failed` Grafana alert matches.
Failed(anyhow::Error),
}
@ -604,6 +604,8 @@ pub fn emit_turn_end(bus: &Bus, outcome: &TurnOutcome) {
ok: false,
note: Some(note.clone()),
});
// The `hh-agent-turn-failed` alert in swarm-grafana.nix matches
// this message exactly.
tracing::warn!(error = %note, "turn failed");
}
}