vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

agents: remove send(operator) and the operator inbox; alert on turn failures

Browse source 
Agents reach the operator over matrix now (#5115). A `send` to
`operator` is refused, by the broker and by the MCP allow-list check,
with an error that names the matrix MCP tools. The dashboard's Y3R C4LL
tab goes with it: the operator inbox, `GET /api/operator-inbox`,
`Broker::unread_for_recipient`, and the browser notifications that only
ever fired on operator-bound messages.

The harness no longer posts turn failures and plugin-install failures
to the operator inbox. Both are already logged at warn; a new Grafana
rule, `hh-agent-turn-failed`, fires per agent on the harness's
`turn failed` line.

Dashboard-to-agent messaging (op-send, the agent web UI's message box,
mark-all-read for agent inboxes) is unchanged.

Decisions per mara on #5115: q1 a, q2 a plus a grafana alert, q3 keep
dashboard-to-agent messaging and leave swarm-level code alone.
This commit is contained in:
atlas 2026-10-12 00:44:11 +02:00 • committed by mara
commit 7e6a4c7727
37 changed files with 246 additions and 848 deletions
Download patch file Download diff file Expand all files Collapse all files

8
hive-agent/prompts/system.md
View file

@ -1,8 +1,8 @@
You are hyperhive agent `{label}` (qualified: `{qualified_label}`){hive_identity}{swarm_identity} in a multi-agent system. The operator (recipient `operator` in `send`, the human at the dashboard) uses **{operator_pronouns}** pronouns — use them naturally when you refer to them in third person (e.g. when relaying to a peer or the manager). When you're talking to or about a peer on a different hive, use the qualified form (`name@hive`) so the operator + peers can disambiguate; within your own hive the short form is fine.
You are hyperhive agent `{label}` (qualified: `{qualified_label}`){hive_identity}{swarm_identity} in a multi-agent system. The operator (the human at the dashboard) uses **{operator_pronouns}** pronouns — use them naturally when you refer to them in third person (e.g. when relaying to a peer or the manager). When you're talking to or about a peer on a different hive, use the qualified form (`name@hive`) so the operator + peers can disambiguate; within your own hive the short form is fine.
Tools (hyperhive surface). Full signature + behavior for each comes from the tool's own MCP description (you already received it via the MCP tool schema) — this is just the map of what exists and which ones are gated, so you know where to look:
- **Inbox / messaging** (always available): `mcp__hyperhive__recv`, `mcp__hyperhive__ack_until`, `mcp__hyperhive__send`, `mcp__hyperhive__get_loose_ends`, `mcp__hyperhive__cancel_loose_end`, `mcp__hyperhive__mark_todos_done`, `mcp__hyperhive__remind`, `mcp__hyperhive__set_status`, `mcp__hyperhive__get_agent_meta`. One habit worth internalizing beyond the tool descriptions themselves: prefer ending the turn over repeatedly polling `recv` when idle (only turn-boundaries observe in-container todo wakes — bash-task completions, matrix unread, forge activity — and ending the turn is also your checkpoint). For a large todo backlog (`get_loose_ends` caps at 40 rows), clear reviewed ids in bulk with `mark_todos_done` rather than cancelling one at a time — there's no blind range-clear, only ids you've actually looked at.
- **Inbox / messaging** (always available): `mcp__hyperhive__recv`, `mcp__hyperhive__ack_until`, `mcp__hyperhive__send`, `mcp__hyperhive__get_loose_ends`, `mcp__hyperhive__cancel_loose_end`, `mcp__hyperhive__mark_todos_done`, `mcp__hyperhive__remind`, `mcp__hyperhive__set_status`, `mcp__hyperhive__get_agent_meta`. `send` reaches peers on this hive only; **reach the operator over matrix** (the matrix MCP tools, `mcp__matrix__send_dm` or `mcp__matrix__send_message`) — `send(to: "operator")` is refused. One habit worth internalizing beyond the tool descriptions themselves: prefer ending the turn over repeatedly polling `recv` when idle (only turn-boundaries observe in-container todo wakes — bash-task completions, matrix unread, forge activity — and ending the turn is also your checkpoint). For a large todo backlog (`get_loose_ends` caps at 40 rows), clear reviewed ids in bulk with `mark_todos_done` rather than cancelling one at a time — there's no blind range-clear, only ids you've actually looked at.
- **Extra MCP tools** (some agents only): `mcp__<server>__<tool>` — agent-specific (matrix client, scraper, db connector, etc.) declared in your `agent.nix` under `hyperhive.extraMcpServers`. First-class tools, already operator-approved at deploy time.
Need new packages, env vars, or other NixOS config for yourself? You can't edit your own config directly — open a PR on your `agent-configs/<name>` repo, or contact the operator directly. Config repos live at `/agents/{label}/config/` (read-only inside your container). All changes flow through operator-approved commits.
@ -13,8 +13,8 @@ Operator boundary: starting, stopping and rebuilding containers is the operator'
Messages from sender `system` are hyperhive helper events (JSON body, `event` field discriminates): `container_crash`, `needs_update`. Use these to react to lifecycle changes:
- `needs_update` — agent's flake rev is stale. Ask the operator to rebuild it.
- `container_crash` — ask the operator to start it again; if it keeps crashing, say so with what you saw.
- `needs_update` — agent's flake rev is stale. Ask the operator over matrix to rebuild it.
- `container_crash` — ask the operator over matrix to start it again; if it keeps crashing, say so with what you saw.
Lifecycle notices that don't need an immediate turn — a new agent spawned, a container rebuilt/killed/destroyed, or its login state changing — surface as todos instead of messages now. Call `get_loose_ends` to see them.

33
hive-agent/src/identity.rs
View file

@ -48,20 +48,9 @@ pub fn swarm_name() -> Option<String> {
non_empty_env("HYPERHIVE_SWARM_NAME")
}
/// Hive-qualified agent identity. When the hive domain is configured, returns
/// `${label}@${domain}` (e.g. `iris@darkest.space`); when not, returns just
/// the short label so callers can render a single string regardless of
/// deployment shape. Callers that want to know whether the result is
/// qualified should check [`hive_domain`] directly.
#[must_use]
pub fn qualified_label() -> String {
qualify(&label())
}
/// Apply hive qualification to an arbitrary agent label (e.g. a peer name
/// from the broker). Mirrors [`qualified_label`] but lets a caller qualify
/// names it didn't read from the env. Returns `${label}@${domain}` when the
/// hive domain is set, else just `label`.
/// Apply hive qualification to an agent label (e.g. a peer name from the
/// broker). Returns `${label}@${domain}` when the hive domain is set, else
/// just `label`.
#[must_use]
pub fn qualify(label: &str) -> String {
match hive_domain() {
@ -151,27 +140,27 @@ mod tests {
}
#[test]
fn qualified_label_with_domain_set() {
fn qualify_with_domain_set() {
with_env(Some("iris"), Some("darkest.space"), || {
assert_eq!(qualified_label(), "iris@darkest.space");
assert_eq!(qualify(&label()), "iris@darkest.space");
assert_eq!(hive_domain().as_deref(), Some("darkest.space"));
});
}
#[test]
fn qualified_label_falls_back_to_short_when_domain_unset() {
fn qualify_falls_back_to_short_when_domain_unset() {
with_env(Some("iris"), None, || {
assert_eq!(qualified_label(), "iris");
assert_eq!(qualify(&label()), "iris");
assert!(hive_domain().is_none());
});
}
#[test]
fn qualified_label_falls_back_to_short_when_domain_empty() {
fn qualify_falls_back_to_short_when_domain_empty() {
// Empty string is treated the same as unset — a misconfigured
// module shouldn't surface `iris@` (no domain) to the operator.
with_env(Some("iris"), Some(""), || {
assert_eq!(qualified_label(), "iris");
assert_eq!(qualify(&label()), "iris");
assert!(hive_domain().is_none());
});
}
@ -247,8 +236,8 @@ mod tests {
assert!(hive_domain().is_none());
assert_eq!(hive_name().as_deref(), Some("pr1ma"));
assert_eq!(swarm_name().as_deref(), Some("constellat1on"));
// qualified_label still degrades to short label without domain.
assert_eq!(qualified_label(), "iris");
// qualify still degrades to the short label without a domain.
assert_eq!(qualify(&label()), "iris");
},
);
}

62
hive-agent/src/main.rs
View file

@ -137,22 +137,6 @@ fn log_system_event(bus: &Bus, from: &str, body: &str) {
});
}
/// Body string for the turn-failure notification we route to the
/// operator on `TurnError::Failed`. Reads the hive-qualified
/// identity so the receiver sees `agent@hive` rather than relying on
/// the caller threading a `label` through every turn-handling layer.
/// Falls back to `<unknown>` when `HIVE_LABEL` is missing so a
/// misconfigured harness still produces a parseable line.
fn format_turn_failure(err: &anyhow::Error) -> String {
let who = crate::identity::qualified_label();
let who = if who.is_empty() {
"<unknown>".to_owned()
} else {
who
};
format!("[system] `{who}` claude turn failed:\n{err:#}")
}
/// What a finished turn tells the serve loop to do next.
struct TurnControl {
/// The turn ended in `AuthFailed` — caller parks on login.
@ -293,16 +277,6 @@ trait Surface {
/// fallback. Same shape as `graceful_stop_complete`.
fn pause_acknowledged(socket: &Path) -> impl Future<Output = ()>;
/// Send a message addressed to the operator. The reporting line out
/// of a container: this is where a turn failure or a plugin-install
/// failure surfaces when nothing inside the harness can act on it.
///
/// Was `<parent>` while the hierarchy existed, a sentinel the broker resolved per
/// `topology.json` and which already fell through to `operator` for a
/// root agent. With the parent field gone every agent takes that
/// branch, so the recipient is written out rather than resolved.
fn send_to_operator(socket: &Path, body: String) -> impl Future<Output = ()>;
/// Long-poll the broker for the next message. Wraps the
/// `Messages`/empty/error trichotomy in `RecvOutcome` so the
/// generic `serve_loop` doesn't need the per-role Response enum
@ -387,22 +361,6 @@ impl Surface for AgentSurface {
(threads, reminders)
}
async fn send_to_operator(socket: &Path, body: String) {
let res = hive_sock_client::request::<_, Response>(
socket,
&Request::Send {
to: hive_sh4re::manager::OPERATOR_RECIPIENT.into(),
body,
in_reply_to: None,
},
CONTROL_SOCKET_RETRY,
)
.await;
if let Err(e) = res {
tracing::warn!(error = ?e, "failed to notify the operator of turn failure");
}
}
async fn recv_next(socket: &Path) -> RecvOutcome {
let recv: Result<Response> = hive_sock_client::request_within(
socket,
@ -548,11 +506,7 @@ async fn serve_main<S: Surface>(socket: &Path, poll_ms: u64) -> Result<()> {
}
}
let files = turn::TurnFiles::prepare(socket, &label).await?;
// Plugin install failures come back as a Vec<String> — route each
// through the `send_to_operator` failure-notify path.
for failure in plugins::install_configured().await {
S::send_to_operator(socket, failure).await;
}
plugins::install_configured().await;
// The forge notification poller used to be spawned here. It is its own
// process now (`hive-forge-notify`, its own systemd unit) so a harness
// restart doesn't take forge notifications down with it; it reaches the
@ -944,7 +898,8 @@ fn apply_todo_wake_checked(checked: Option<bool>, todo_miss_streak: &mut u32, bu
}
/// Drive a single turn: emit boot-of-turn events, run claude, ack on
/// success / requeue on rate-limit-or-401 / notify parent on failure,
/// success / requeue on rate-limit-or-401 / log `turn failed` on a hard
/// failure (the line the `hh-agent-turn-failed` Grafana rule alerts on),
/// record stats. Returns a `TurnControl` carrying the auth-failed flag and
/// the todo-wake-checked signal — the serve loop decides what to do next.
async fn handle_turn<S: Surface>(
@ -1061,9 +1016,11 @@ async fn handle_turn<S: Surface>(
/// The non-happy-path half of `handle_turn`: react to each `TurnError`
/// variant the turn could have failed with (park-and-retry on rate-limit/
/// stall/auth, requeue-for-a-fresh-turn on prompt-too-long/session-not-
/// found, notify the operator on a hard failure). Split out purely to keep
/// `handle_turn` itself under clippy's line-count lint — no behavior
/// change from when this lived inline.
/// found). A hard failure needs nothing here: `turn::emit_turn_end` has
/// already logged `turn failed`, which the `hh-agent-turn-failed` Grafana
/// rule alerts on. Split out purely to keep `handle_turn` itself under
/// clippy's line-count lint — no behavior change from when this lived
/// inline.
async fn handle_turn_error_recovery<S: Surface>(
outcome: &turn::TurnOutcome,
bus: &Bus,
@ -1123,7 +1080,4 @@ async fn handle_turn_error_recovery<S: Surface>(
tracing::warn!("session-not-found; requeueing message for a fresh turn");
S::requeue_inflight(socket).await;
}
if let Err(turn::TurnError::Failed(e)) = outcome {
S::send_to_operator(socket, format_turn_failure(e)).await;
}
}

22
hive-agent/src/plugins.rs
View file

@ -96,23 +96,20 @@ async fn update_marketplaces() {
}
/// Install every plugin in `/etc/hyperhive/claude-plugins.json`.
/// Returns a list of human-readable failure messages so the caller can
/// route them through their own per-role surface (turn-failure-style
/// notification, see `Surface::send_to_operator`). Wire-agnostic: the
/// caller picks the recipient, the same way failure-notify does.
pub async fn install_configured() -> Vec<String> {
/// A failed install is logged and skipped; the harness starts without it.
pub async fn install_configured() {
let Ok(raw) = tokio::fs::read_to_string(PLUGINS_PATH).await else {
return Vec::new();
return;
};
let specs: Vec<String> = match serde_json::from_str(&raw) {
Ok(v) => v,
Err(e) => {
tracing::warn!(path = PLUGINS_PATH, error = ?e, "claude-plugins spec parse failed; skipping");
return Vec::new();
return;
}
};
if specs.is_empty() {
return Vec::new();
return;
}
add_marketplaces().await;
if auto_update_enabled().await {
@ -120,7 +117,6 @@ pub async fn install_configured() -> Vec<String> {
} else {
tracing::debug!("claudePluginsAutoUpdate=false, skipping marketplace update");
}
let mut failures = Vec::new();
for spec in specs {
match Command::new("claude")
.args(["plugin", "install", &spec])
@ -138,18 +134,10 @@ pub async fn install_configured() -> Vec<String> {
stderr = %stderr,
"claude plugin install failed",
);
failures.push(format!(
"claude plugin install failed for `{spec}`:\n{}",
stderr.trim()
));
}
Err(e) => {
tracing::warn!(spec = %spec, error = ?e, "claude plugin install spawn failed");
failures.push(format!(
"claude plugin install spawn failed for `{spec}`: {e}"
));
}
}
}
failures
}

6
hive-agent/src/turn.rs
View file

@ -201,8 +201,8 @@ pub enum TurnError {
/// provider (e.g. HTTP 429) without reporting it ends up here. Handled
/// like [`Self::ApiStall`].
AgentStall(String),
/// A hard failure with no recovery — the serve loop escalates it to the
/// operator (`send_to_operator`).
/// A hard failure with no recovery. Logged as `turn failed`, the exact
/// message the `hh-agent-turn-failed` Grafana alert matches.
Failed(anyhow::Error),
}
@ -604,6 +604,8 @@ pub fn emit_turn_end(bus: &Bus, outcome: &TurnOutcome) {
ok: false,
note: Some(note.clone()),
});
// The `hh-agent-turn-failed` alert in swarm-grafana.nix matches
// this message exactly.
tracing::warn!(error = %note, "turn failed");
}
}