[feat] add CrossDesk MCP with remote desktop panels

2026-11-05 13:42:38 +08:00 · 2026-11-03 02:44:55 +08:00
parent 2fab61871e
commit f57b4cc4e3
40 changed files with 6325 additions and 28 deletions
@@ -0,0 +1,110 @@
+# Native automation protocol
+
+`crossdesk-cli agent --stdio --password-fd 3 --video-ws` is the private native adapter used by CrossDesk MCP. It is separate from the existing terminal-oriented `bridge` protocol. This is **not MCP on stdin**: the TypeScript service owns the public MCP transport and tool schemas.
+
+Optional native arguments are `--server HOST`, `--port PORT`, `--quality PRESET` and `--log-dir DIRECTORY`. The caller creates the log directory. Diagnostic stdout is redirected to stderr; only protocol responses use the preserved output descriptor. Passwords travel over descriptor 3, terminated by newline or EOF, and never appear in JSON requests. `--video-ws` starts an authenticated, native loopback WebSocket listener on an ephemeral port. There is no video file descriptor. Agent mode selects the high-frame-rate preview tier at up to 1920 × 1080 by default; negotiated remote capabilities and network conditions determine the delivered rate. Each process accepts one connection attempt; retry authentication with a new process and credential pipe.
+
+## Envelope
+
+One UTF-8 JSON object per line, maximum request 64 KiB and nesting depth 32. Use nonnegative integer request IDs. Responses may complete out of order; correlate by ID.
+
+```json
+{"id":1,"method":"hello","protocol":"crossdesk-automation/1"}
+{"id":1,"ok":true,"result":{"protocol":"crossdesk-automation/1","image_formats":["image/png"],"action_completion":"submitted","human_pause":true,"max_request_bytes":65536,"max_response_bytes":4194304,"operation_cache_entries":128,"operation_cache_ms":300000}}
+```
+
+Unknown fields and malformed arguments are rejected. Error responses contain a stable code, retryability and whether input may already have been submitted:
+
+```json
+{"id":2,"ok":false,"error":{"code":"CONTROL_DENIED","retryable":false,"side_effect_possible":false}}
+```
+
+The output queue is bounded to 64 messages and 8 MiB; a blocked output pipe terminates after 10 seconds without progress. EOF, signals and `exit` disconnect and release held input. A caller that needs responses must keep stdin open until they arrive.
+
+## Methods
+
+| Method | Parameters in addition to `id`/`method` |
+| --- | --- |
+| `hello` | `protocol: "crossdesk-automation/1"` |
+| `connect` | `device` (9 digits), `mode: "observe" | "control"`, `display` (0–63), optional `decode_path: "browser" / "minirtc"`, `decoder: "auto" / "software"` |
+| `status` | none |
+| `screenshot` | optional `region: {x,y,width,height}`, `max_edge` (64–2048), `after_frame_id`, `generation`, `timeout_ms` (0–5000) |
+| `video_open` | `ticket`, `view`, `view_generation`, `session`, `origin`, `apps`; requires `--video-ws` and a connected session |
+| `video_revoke` | optional `view`; omit to revoke all video grants and sockets |
+| `preview` | optional `max_edge` (64–960), `after_frame_id`, `generation`, `timeout_ms` (0–5000); no region |
+| `action` | `operation_id`, `observation_id`, `action` |
+| `control` | `enabled` boolean; cannot override observe-only permission or human pause |
+| `viewer_control` | `operation: "pause" | "resume" | "disconnect"`; restricted to trusted UI by the MCP adapter |
+| `display` | `index` (0–63) |
+| `cancel` | `request_id`; cancels queued/in-flight captures or input sequences |
+| `disconnect`, `exit` | none; `exit` also ends the process |
+
+`connect` returns immediately after initiating transport. Poll `status` until `state == "ready"`; this requires a frame from the current display generation. Status includes `transport_state`, `geometry_epoch`, `frame_age_ms`, `source_size`, `control_allowed`, `human_paused`, host platform, displays and transport diagnostics.
+
+## Live video
+
+`doctor --json` advertises `automation_video_transport: "loopback-websocket/1"`, `automation_video_protocols: ["h264-annexb/1", "nv12/1"]`, and `automation_video_decoders: ["auto", "software"]`. `hello` repeats these without the `automation_` prefix. The MCP adapter requires the transport capability, so an old pipe-only CLI is rejected. Node spawns only stdin, stdout, stderr, and the credential pipe; live video never enters Node.
+
+`connect.decode_path` defaults to `browser` (compressed H.264), or selects `minirtc` (decoded NV12). `connect.decoder` defaults to `auto` (hardware preferred, software when unavailable); `software` selects the native software decoder. Both paths maintain native screenshot frames with the selected decoder. These choices are fixed for the connection. Status exposes `video` and the reported `native_video_decoder` backend name. PNG encoding runs only when requested. The legacy `preview` image method remains available to private protocol callers but is not used by MCP panels.
+
+### Authorization and transport
+
+The trusted adapter generates 24 random bytes as a 32-character base64url `ticket` and sends `video_open` over private JSON-RPC. `view` and `session` are also 32-character base64url IDs; `view_generation` is positive. `origin` is the Node panel's exact `http://127.0.0.1:PORT` origin. `apps` permits an opaque (`null`) or HTTPS MCP Apps origin, but still requires the same one-time ticket. No password is included.
+
+The response contains `{url, transport, protocol, decoder, expires_in_ms:15000}`. Node supplies the ticket to the authorized UI. The browser opens the returned `ws://127.0.0.1:PORT/api/video` directly and sends `{"ticket":"..."}` as its first text message within 3 seconds. Tickets expire after 15 seconds, are consumed once, and are bound to a view/session plus the native display generation. The native service checks the exact Host and path, RFC 6455 version, and authorized Origin. It does not serve HTTP content or accept remote input commands. Logs are disabled for this listener; tickets never go in URLs or model-visible results.
+
+The listener uses its own I/O thread, at most 16 WebSocket candidates and 8 authorized viewers. Client messages are limited to 1024 bytes. Revocation, disconnect, native generation changes, or shutdown cancel sockets and invalidate tickets, including partially sent frames. Pause only disables Agent control; viewing continues. The adapter explicitly revokes a removed or replaced view, and terminates the native session if revocation cannot be confirmed. Native processes own ephemeral ports, so panel CSP allows `ws://127.0.0.1:*`, restricted to the literal loopback address; the control HTTP origin remains exact.
+
+### Binary messages and flow control
+
+Each binary WebSocket message contains one complete frame, **without the old pipe length prefix**. Integers are big-endian. For `h264-annexb/1`:
+
+| Offset | Type | Value |
+| --- | --- | --- |
+| 0 | u32 | `0x43445631` (`CDV1`) |
+| 4 | u32 | Display/connection generation |
+| 8 | u32 | Receive sequence; gaps require a new keyframe |
+| 12 | u64 | Local monotonic receive timestamp in microseconds |
+| 20 | u32 | Flags: 1 = IDR keyframe, 0 = delta |
+| 24 | bytes | Annex B H.264; keyframes include SPS and PPS |
+
+Access units are capped at 4 MiB. Both the receive tap and the I/O handoff are bounded to 60 frames / 8 MiB. Overflow discards dependent frames until an IDR. After receiving each message the browser sends `{"type":"ack","generation":N,"sequence":N}` directly to the native socket, in order. This acknowledges receipt, not decode, presentation, or remote execution. Per-viewer unacknowledged output is capped at 8 frames; at 1 MiB outstanding, further frames are skipped until space is available (a single keyframe may exceed that threshold). Dropping a dependent picture requires a new keyframe; `{"type":"keyframe"}` requests recovery, rate-limited to once per second.
+
+For `nv12/1`, magic is `0x43444e31` (`CDN1`). Offsets 4, 8, 12 and 20 remain generation, decoded sequence, local monotonic decoded-frame queue time in microseconds, and flags (always 1). Offsets 24 and 28 are u32 width and height; offset 32 starts tightly packed Y followed by interleaved UV. Dimensions must be even, at most 8192 per axis / 16777216 pixels. Message size must equal `32 + width * height * 3 / 2` (at most 25165856 bytes). Color is limited-range BT.601, matching the existing PNG conversion. Sequence gaps are allowed.
+
+NV12 retains one latest picture in the I/O handoff and at most one unacknowledged picture per viewer. New pictures are skipped for a slow viewer without blocking others. Each prepared WebSocket message is immutable and shared between recipients, avoiding per-viewer payload copies. There is still a native copy into the WebSocket payload and raw pixel transfer to the browser; this is not GPU zero-copy. A viewer whose oldest outstanding frame remains unacknowledged for 10 seconds is disconnected. Hidden/unloaded pages close their native socket. The browser replaces pending raw pictures and does not decode H.264 in this mode.
+
+`status.video_output` includes `transport`, `viewers`, I/O handoff `queued` / `queued_bytes`, aggregate `unacknowledged_bytes`, and `dropped` (handoff replacements plus per-viewer congestion skips). These diagnostics contain no video payload. The original receive-tap SPS/PPS and keyframe state remain available for H.264.
+
+The MiniRTC `Params` structure appends `on_receive_encoded_video`, changing its ABI size. Rebuild consumers and MiniRTC together; zero-initialize `Params`. The callback borrows frame bytes on the receive/decode queue and must not retain the pointer. Returning nonzero requests a remote keyframe. No remote wire-protocol change is required.
+
+## Images and observations
+
+Images return `{mime_type:"image/png",data:"<base64>",metadata:{...}}`. Metadata includes `generation`, `geometry_epoch`, `frame_id`, `display_index`, `source_size`, `source_rect`, `image_size` and `frame_age_ms`; only screenshots create `observation_id`. Preview images cannot authorize model input.
+
+NV12 frames must be tightly packed, even dimensions, at most 8192 per axis and 16 million pixels. Conversion/PNG encoding runs on one worker with no `ControllerSession` calls. The owner loop continues processing cancellation and input release. Screenshots have priority over queued previews. An encoding from an obsolete generation or geometry epoch is discarded. Frames older than two seconds are not used for a new image. PNG limits are 2 MiB for observations and 256 KiB for previews; the encoder reduces dimensions when necessary.
+
+`after_frame_id` requires `generation`. A screenshot waits up to `timeout_ms` for a newer frame; a preview can return `{unchanged:true, frame_age_ms, stale, generation, frame_id}`. Neither path treats an image as an input acknowledgement.
+
+## Actions
+
+All coordinates refer to the immutable returned observation image. The native layer maps them through its source rectangle to normalized remote coordinates, without terminal layout padding. Coordinates must satisfy `0 <= x < image_width` and `0 <= y < image_height`. Observations expire after 30 seconds, retain at most 32 entries, and are invalidated by geometry/display generation changes.
+
+```json
+{"id":10,"method":"action","operation_id":"click-1","observation_id":"obs_2_1","action":{"type":"click","x":120,"y":80,"button":"left"}}
+```
+
+Supported payloads:
+
+- `move`: `x`, `y`.
+- `click`, `double_click`: `x`, `y`, optional `button` (`left`, `right`, `middle`).
+- `scroll`: `x`, `y`, `direction` (`up`, `down`, `left`, `right`), optional `ticks` (1–10).
+- `drag`: `path` of 2–128 `{x,y}` points, optional `duration_ms` (100–2000).
+- `key_press`: `key`; `hotkey`: 2–6 unique `keys`.
+- `type_text`: `text`, at most 16 KiB UTF-8; requires negotiated remote text-input capability.
+
+Only one input sequence runs at once. Timed steps are advanced by the owner loop; cancellation, disconnect, display changes and pause release held buttons/keys. A failed send is conservatively considered potentially submitted.
+
+Results use `status: submitted | in_progress | failed_before_submit | outcome_unknown`, with `operation_id`, `retryable`, `side_effect_possible` and, for completed actions, `generation`/`after_frame_id`. The same operation ID and arguments retrieve the cached result; different arguments conflict. Retention is bounded to 128 entries, with entries older than five minutes eligible for removal on a new action. Never retry an uncertain effect with a new ID. `submitted` does not claim remote execution.
+
+The native process trusts its parent. The parent must authenticate panel-only `viewer_control` calls and protect descriptor 3; this private pipe does not isolate arbitrary programs running as the same OS user.
@@ -7,6 +7,8 @@ CLI 从独立开发目录整合到 CrossDesk，入口、打包脚本、npm 启
 - `ControllerSession` 的流名称、HostInformation 和输入参考 Android `rtc_bridge.cpp`；身份使用单 Peer 的 `cli` 临时注册协议；终端适配不依赖 JNI、Surface 或 GuiRuntime。
 - 帧缓存遵守 `MiniRtcVideoFrame` 回调借用契约，直接复制到有界 CPU 存储，不跨回调保留 native 指针。
 - PreviewSubscription 复用 wire 实现与对端 capability 字段，不新增被控端协议。
+- MCP 实时面板新增 MiniRTC 的 `on_receive_encoded_video` 回调，转发解码前的完整 H.264 Annex B 帧；回调及关键帧请求由当前子模块源码提供。`Params` 新增字段改变 ABI 大小，CLI 与 MiniRTC 必须一起重建；发布时同步记录包含该改动的子模块提交。
+- 自动化图像编码复用 MiniRTC 已有的 libyuv 2025.8.14，并通过 Xmake 固定静态 libpng v1.6.59；libpng 的压缩依赖沿用平台 zlib。新增依赖的许可证和对应源码也应纳入正式发行的依赖材料。

 MiniRTC 已在提交 `b9ee1fa7fe038fdf771f5dec1b1d561d78273f15` 中合入凭据日志脱敏：Join、排队 Join、取消 Join、Leave 四个日志点只记录 `@` 前的 ID，不记录密码。`apps/cli/scripts/prepare-deps.sh` 仅按主仓库记录初始化子模块，不再应用独立补丁；更新 MiniRTC 时只需更新子模块提交。共享子模块中的脱敏同样适用于桌面程序；该修改没有改变鉴权、TLS、TURN 或媒体协议。

@@ -183,6 +183,6 @@ Pi 扩展读取宿主的字符尺寸，并通过宿主终端写入／输入监
 - `doctor --json` 是环境报告，不会联网登录、主动探测 Kitty / Sixel 或证明远端可连接。 `input_shortcuts=local_commands_v2` 表示支持本文的持续本地命令模式；启动日志同时记录实际执行路径，以便区分不同工程的构建产物。
 - 画面比例异常时查看 `client.log` 摘要的 `terminal_geometry`：`cell_aspect` 为实际采用的字符宽高比，`reported` 表示收到有效的终端尺寸报告，`manual_aspect` 表示使用了手动覆盖。`doctor` 中的比例和像素尺寸只反映系统 TTY 信息，不主动查询终端，可能与交互连接自动检测后的值不同。

-第一轮无音频播放或音频订阅 API，SSH 服务器上的声音不会传回本地。无后台守护进程、共享媒体 Broker、IP 直连、MCP 或自动截图。其他宿主可以使用 [私有桥接](bridge-protocol.md)，没有可靠自定义 UI 时打开独立终端或给出播放器命令。
+终端模式无音频播放或音频订阅 API，SSH 服务器上的声音不会传回本地。无后台守护进程、共享媒体 Broker 或 IP 直连。终端宿主可以使用 [私有桥接](bridge-protocol.md)，没有可靠自定义 UI 时打开独立终端。Agent 的 PNG 截图和结构化操作由独立的 [自动化模式](automation-protocol.md) 和 [MCP 服务](../../mcp/README.md) 提供。

 每个播放器/桥接进程只拥有自己的一个媒体会话，不向模型消息发送视频。