vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

feat(#2305): hive-screen-mcp — screenshot + input MCP for GUI agents

Browse source 
New crate hive-screen-mcp: a stdio MCP bridge activated automatically
when an agent has hyperhive.gui.enable = true. Provides five tools:

- screenshot   — grim → saves PNG, returns path for Read tool
- type_text    — wtype → Unicode text input (no daemon)
- key_press    — ydotool key → combos like ctrl+c, super+l
- mouse_move   — ydotool mousemove --absolute
- mouse_click  — ydotool click, optionally with prior move

New nix/agent-modules/screen.nix: wires the MCP bridge into
extraMcpServers.screen; adds grim + wtype to systemPackages. Adds
hyperhive.gui.screenInput option (default false) which enables the
ydotoold daemon + ydotool for mouse/keyboard injection via /dev/uinput.

screenshot and type_text work without screenInput. key_press,
mouse_move, and mouse_click return a ydotool error until ydotoold is
running and /dev/uinput is accessible in the container.
This commit is contained in:
iris 2026-07-20 20:21:38 +02:00 • committed by mara
commit 0b3268feae
8 changed files with 311 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

209
hive-screen-mcp/src/main.rs Normal file
View file

@ -0,0 +1,209 @@
//! `hive-screen-mcp` — stdio MCP bridge for GUI agents.
//!
//! Provides screenshot, text input, and mouse/keyboard tools for agents
//! running a Weston Wayland compositor (`hyperhive.gui.enable = true`).
//! Each tool shells out to the appropriate utility:
//!
//! - `screenshot` → `grim` (always available with gui enabled)
//! - `type_text` → `wtype` (Wayland-native text input, no daemon)
//! - `key_press` → `ydotool key` (needs ydotoold + `/dev/uinput`)
//! - `mouse_move` → `ydotool mousemove` (same requirement)
//! - `mouse_click` → `ydotool click` (same requirement)
//!
//! `WAYLAND_DISPLAY` and `XDG_RUNTIME_DIR` are injected globally by the
//! `weston-vnc` module so all tools find the compositor automatically.
//! `ydotool`-based tools require the ydotoold daemon; enable it by
//! setting `hyperhive.gui.screenInput = true` in the agent config.
use anyhow::Result;
use rmcp::{
ServerHandler, ServiceExt,
handler::server::wrapper::Parameters,
schemars::{self, JsonSchema},
tool, tool_handler, tool_router,
transport::stdio,
};
use serde::Deserialize;
use tokio::process::Command;
/// Run a command and return a human-readable result string.
/// On success returns stdout (trimmed) or `"ok"` when stdout is empty.
/// On failure returns a `"command failed (exit N): <stderr>"` string.
async fn run_cmd(program: &str, args: &[&str]) -> String {
match Command::new(program).args(args).output().await {
Ok(out) if out.status.success() => {
let s = String::from_utf8_lossy(&out.stdout).trim().to_owned();
if s.is_empty() { "ok".to_owned() } else { s }
}
Ok(out) => {
let msg = String::from_utf8_lossy(&out.stderr).trim().to_owned();
format!(
"command failed (exit {}): {}",
out.status.code().unwrap_or(-1),
if msg.is_empty() { "(no stderr)" } else { &msg }
)
}
Err(e) => format!("failed to launch `{program}`: {e}"),
}
}
struct ScreenMcp;
#[tool_router]
impl ScreenMcp {
#[tool(
description = "Take a screenshot of the agent's Wayland display and save it as a PNG \
file. Returns the path to the saved file — pass that path to the `Read` tool to \
view the image visually. `dest_path` is optional; when omitted a timestamped temp \
file under `/tmp/` is used. Requires `grim` and a live `WAYLAND_DISPLAY` \
(provided automatically when `hyperhive.gui.enable = true`)."
)]
async fn screenshot(&self, Parameters(args): Parameters<ScreenshotArgs>) -> String {
let path = args.dest_path.unwrap_or_else(|| {
let ms = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map_or(0, |d| d.as_millis());
format!("/tmp/hive-screenshot-{ms}.png")
});
let result = run_cmd("grim", &["-t", "png", &path]).await;
if result == "ok" {
format!("screenshot saved to `{path}` — use the Read tool to view it")
} else {
result
}
}
#[tool(
description = "Type text into the currently focused Wayland window. Uses `wtype`, \
which supports arbitrary Unicode. For special keys (Enter, Tab, Escape, function \
keys, arrow keys) or modifier combos, use `key_press` instead. Requires a live \
`WAYLAND_DISPLAY` (provided automatically when `hyperhive.gui.enable = true`)."
)]
async fn type_text(&self, Parameters(args): Parameters<TypeTextArgs>) -> String {
run_cmd("wtype", &[&args.text]).await
}
#[tool(
description = "Press a key or key combination. `keys` uses ydotool syntax: a single \
key name (`Return`, `Tab`, `Escape`, `BackSpace`, `Delete`, `space`, `F1`–`F12`, \
`Left`, `Right`, `Up`, `Down`) or a modifier+key combo (`ctrl+c`, \
`ctrl+shift+t`, `super+l`, `alt+F4`). Requires the ydotoold daemon and \
`/dev/uinput` device access in the container — set \
`hyperhive.gui.screenInput = true` in the agent config to enable the daemon."
)]
async fn key_press(&self, Parameters(args): Parameters<KeyPressArgs>) -> String {
run_cmd("ydotool", &["key", &args.keys]).await
}
#[tool(
description = "Move the mouse cursor to an absolute pixel position on the display. \
`x` is measured from the left edge, `y` from the top edge. Requires the ydotoold \
daemon and `/dev/uinput` device access — set `hyperhive.gui.screenInput = true` \
in the agent config to enable the daemon."
)]
async fn mouse_move(&self, Parameters(args): Parameters<MouseMoveArgs>) -> String {
run_cmd(
"ydotool",
&[
"mousemove",
"--absolute",
"-x",
&args.x.to_string(),
"-y",
&args.y.to_string(),
],
)
.await
}
#[tool(
description = "Click a mouse button, optionally moving to a position first. `button` \
is `\"left\"` (default), `\"right\"`, or `\"middle\"`. When `x` and `y` are \
provided the cursor is moved to that absolute pixel position before clicking. \
Requires the ydotoold daemon and `/dev/uinput` device access — set \
`hyperhive.gui.screenInput = true` in the agent config to enable the daemon."
)]
async fn mouse_click(&self, Parameters(args): Parameters<MouseClickArgs>) -> String {
// Move first when coordinates are supplied.
if let (Some(x), Some(y)) = (args.x, args.y) {
let mv = run_cmd(
"ydotool",
&[
"mousemove",
"--absolute",
"-x",
&x.to_string(),
"-y",
&y.to_string(),
],
)
.await;
if mv != "ok" {
return mv;
}
}
// Map the button name to ydotool's button code.
let button = match args.button.as_deref().unwrap_or("left") {
"right" => "0xC1",
"middle" => "0xC2",
_ => "0xC0", // left
};
run_cmd("ydotool", &["click", button]).await
}
}
#[derive(Debug, Deserialize, JsonSchema)]
struct ScreenshotArgs {
/// Optional destination path for the PNG file. When omitted a
/// timestamped path under `/tmp/` is generated automatically.
dest_path: Option<String>,
}
#[derive(Debug, Deserialize, JsonSchema)]
struct TypeTextArgs {
/// Text to type into the focused window. Supports arbitrary Unicode.
text: String,
}
#[derive(Debug, Deserialize, JsonSchema)]
struct KeyPressArgs {
/// Key or key combination in ydotool syntax — e.g. `"Return"`,
/// `"Tab"`, `"ctrl+c"`, `"ctrl+shift+t"`, `"super+l"`, `"alt+F4"`.
keys: String,
}
#[derive(Debug, Deserialize, JsonSchema)]
struct MouseMoveArgs {
/// Horizontal pixel coordinate from the left edge of the display.
x: i32,
/// Vertical pixel coordinate from the top edge of the display.
y: i32,
}
#[derive(Debug, Deserialize, JsonSchema)]
struct MouseClickArgs {
/// Button to click: `"left"` (default), `"right"`, or `"middle"`.
button: Option<String>,
/// Optional X coordinate to move to before clicking (absolute pixels).
x: Option<i32>,
/// Optional Y coordinate to move to before clicking (absolute pixels).
y: Option<i32>,
}
#[tool_handler]
impl ServerHandler for ScreenMcp {}
#[tokio::main]
async fn main() -> Result<()> {
tracing_subscriber::fmt()
.with_env_filter(
tracing_subscriber::EnvFilter::try_from_env("RUST_LOG")
.unwrap_or_else(|_| tracing_subscriber::EnvFilter::new("warn")),
)
.with_writer(std::io::stderr)
.init();
let service = ScreenMcp.serve(stdio()).await?;
service.waiting().await?;
Ok(())
}