feat(#2618): add mouse_move + mouse_click via RFB PointerEvent
Implements mouse input by speaking the RFB protocol directly to Weston's neatvnc server (localhost:HIVE_GUI_VNC_PORT, default 5900) — the VNC backend's native remote-input path. No /dev/uinput, no kernel bypass; the compositor mediates all input just as it does for the browser VNC viewer. Changes: - rfb_handshake(): RFB 3.8 handshake with security type None (auth-method=none in weston.ini); shared-session ClientInit keeps the browser viewer connected - rfb_pointer_event(): encodes a 6-byte RFB PointerEvent (type=5, button-mask, x/y big-endian) - rfb_send_pointer_events(): connects, handshakes, sends an event slice, flushes — all in one TCP connection - mouse_move(x, y): sends a single PointerEvent(mask=0, x, y) - mouse_click(x, y, button): sends move → button-down → button-up sequence (left/middle/right via RFB button-mask bits 0/1/2) - vnc_port(): reads HIVE_GUI_VNC_PORT from env, falls back to 5900 No new packages or nix options — HIVE_GUI_VNC_PORT is already set by the harness when gui.enable = true; grim/wtype are the only runtime deps. Closes #2618.
This commit is contained in:
parent
804c11c404
commit
228a5bacca
2 changed files with 209 additions and 14 deletions
|
|
@ -1,13 +1,15 @@
|
|||
# Screen MCP — screenshot + keyboard input for GUI agents.
|
||||
# Screen MCP — screenshot, keyboard, and mouse for GUI agents.
|
||||
#
|
||||
# Auto-activated when `hyperhive.gui.enable = true`. Wires the
|
||||
# `hive-screen-mcp` stdio bridge as `extraMcpServers.screen` so claude
|
||||
# gets three tools: `screenshot`, `type_text`, and `key_press`.
|
||||
# gets five tools: `screenshot`, `type_text`, `key_press`,
|
||||
# `mouse_move`, and `mouse_click`.
|
||||
#
|
||||
# All tools use Wayland protocols mediated by the Weston compositor —
|
||||
# no `/dev/uinput` or kernel-level injection needed. `grim` takes
|
||||
# screenshots; `wtype` handles both text input and key/modifier combos
|
||||
# via the `zwp-virtual-keyboard-unstable-v1` protocol.
|
||||
# All tools operate in userspace without `/dev/uinput`:
|
||||
# - `grim` takes screenshots via Wayland screencopy
|
||||
# - `wtype` handles text + key combos via `zwp-virtual-keyboard-unstable-v1`
|
||||
# - mouse tools speak RFB `PointerEvent` directly to the local neatvnc
|
||||
# server (port `HIVE_GUI_VNC_PORT`) — the VNC backend's native input path
|
||||
{
|
||||
pkgs,
|
||||
lib,
|
||||
|
|
|
|||
Loading…
Reference in a new issue