228a5bacca
Implements mouse input by speaking the RFB protocol directly to Weston's neatvnc server (localhost:HIVE_GUI_VNC_PORT, default 5900) — the VNC backend's native remote-input path. No /dev/uinput, no kernel bypass; the compositor mediates all input just as it does for the browser VNC viewer. Changes: - rfb_handshake(): RFB 3.8 handshake with security type None (auth-method=none in weston.ini); shared-session ClientInit keeps the browser viewer connected - rfb_pointer_event(): encodes a 6-byte RFB PointerEvent (type=5, button-mask, x/y big-endian) - rfb_send_pointer_events(): connects, handshakes, sends an event slice, flushes — all in one TCP connection - mouse_move(x, y): sends a single PointerEvent(mask=0, x, y) - mouse_click(x, y, button): sends move → button-down → button-up sequence (left/middle/right via RFB button-mask bits 0/1/2) - vnc_port(): reads HIVE_GUI_VNC_PORT from env, falls back to 5900 No new packages or nix options — HIVE_GUI_VNC_PORT is already set by the harness when gui.enable = true; grim/wtype are the only runtime deps. Closes #2618.
34 lines
1.1 KiB
Nix
34 lines
1.1 KiB
Nix
# Screen MCP — screenshot, keyboard, and mouse for GUI agents.
|
|
#
|
|
# Auto-activated when `hyperhive.gui.enable = true`. Wires the
|
|
# `hive-screen-mcp` stdio bridge as `extraMcpServers.screen` so claude
|
|
# gets five tools: `screenshot`, `type_text`, `key_press`,
|
|
# `mouse_move`, and `mouse_click`.
|
|
#
|
|
# All tools operate in userspace without `/dev/uinput`:
|
|
# - `grim` takes screenshots via Wayland screencopy
|
|
# - `wtype` handles text + key combos via `zwp-virtual-keyboard-unstable-v1`
|
|
# - mouse tools speak RFB `PointerEvent` directly to the local neatvnc
|
|
# server (port `HIVE_GUI_VNC_PORT`) — the VNC backend's native input path
|
|
{
|
|
pkgs,
|
|
lib,
|
|
config,
|
|
...
|
|
}:
|
|
{
|
|
config = lib.mkIf config.hyperhive.gui.enable {
|
|
# Register the screen MCP bridge so claude gets the screen tools.
|
|
hyperhive.extraMcpServers.screen = {
|
|
command = "${config.hyperhive.packages.hive-screen-mcp}/bin/hive-screen-mcp";
|
|
args = [ ];
|
|
};
|
|
|
|
# grim: Wayland screenshot; wtype: text + key input via virtual-keyboard
|
|
# protocol (compositor-mediated, no /dev/uinput required).
|
|
environment.systemPackages = [
|
|
pkgs.grim
|
|
pkgs.wtype
|
|
];
|
|
};
|
|
}
|