feat(#2618): add mouse_move + mouse_click via RFB PointerEvent

Implements mouse input by speaking the RFB protocol directly to Weston's
neatvnc server (localhost:HIVE_GUI_VNC_PORT, default 5900) — the VNC
backend's native remote-input path. No /dev/uinput, no kernel bypass;
the compositor mediates all input just as it does for the browser VNC viewer.

Changes:
- rfb_handshake(): RFB 3.8 handshake with security type None (auth-method=none
  in weston.ini); shared-session ClientInit keeps the browser viewer connected
- rfb_pointer_event(): encodes a 6-byte RFB PointerEvent (type=5, button-mask,
  x/y big-endian)
- rfb_send_pointer_events(): connects, handshakes, sends an event slice,
  flushes — all in one TCP connection
- mouse_move(x, y): sends a single PointerEvent(mask=0, x, y)
- mouse_click(x, y, button): sends move → button-down → button-up sequence
  (left/middle/right via RFB button-mask bits 0/1/2)
- vnc_port(): reads HIVE_GUI_VNC_PORT from env, falls back to 5900

No new packages or nix options — HIVE_GUI_VNC_PORT is already set by the
harness when gui.enable = true; grim/wtype are the only runtime deps.

Closes #2618.
This commit is contained in:
iris 2026-07-20 20:58:13 +02:00
commit 228a5bacca
2 changed files with 209 additions and 14 deletions

View file

@ -1,13 +1,15 @@
# Screen MCP — screenshot + keyboard input for GUI agents.
# Screen MCP — screenshot, keyboard, and mouse for GUI agents.
#
# Auto-activated when `hyperhive.gui.enable = true`. Wires the
# `hive-screen-mcp` stdio bridge as `extraMcpServers.screen` so claude
# gets three tools: `screenshot`, `type_text`, and `key_press`.
# gets five tools: `screenshot`, `type_text`, `key_press`,
# `mouse_move`, and `mouse_click`.
#
# All tools use Wayland protocols mediated by the Weston compositor —
# no `/dev/uinput` or kernel-level injection needed. `grim` takes
# screenshots; `wtype` handles both text input and key/modifier combos
# via the `zwp-virtual-keyboard-unstable-v1` protocol.
# All tools operate in userspace without `/dev/uinput`:
# - `grim` takes screenshots via Wayland screencopy
# - `wtype` handles text + key combos via `zwp-virtual-keyboard-unstable-v1`
# - mouse tools speak RFB `PointerEvent` directly to the local neatvnc
# server (port `HIVE_GUI_VNC_PORT`) — the VNC backend's native input path
{
pkgs,
lib,