[fix] stabilize MCP sessions and extend input controls

apps/mcp/README.md

@@ -61,6 +61,8 @@ Windows 和 Linux 使用各自构建产物路径；`--native-binary` 必须指
 
 `crossdesk_open` 立即返回，让 Apps 宿主有机会渲染面板；等待连接在后续 `crossdesk_get_session` 调用中完成。按 `view_id` 查询默认最多等待 25 秒，`wait_ms: 0` 则立即查询；未就绪时返回 `wait_timed_out: true`，Agent 在原任务仍待执行时继续等待。按 `session_id` 查询仍立即返回。取消等待只取消本次查询，不断开用户正在建立或使用的连接。用户主动断开后返回 `closed`，Agent 停止等待，不自动重连。
 
+面板发起的连接失败后，按 `view_id` 等待也会返回 `failed` 和错误码，避免 Agent 一直停留在等待用户状态。断开正在建立的连接会取消整个连接尝试。多个面板和 Agent 同时查询状态时共用一次原生查询，单次查询取消或暂时繁忙不会断开会话。`crossdesk_connect` 在连接成功但面板初始化失败时仍返回有效的 `session_id`，并附加 `panel_error`。
+
 自动继续依赖宿主保持当前 Agent 任务及工具调用流程。它不通过网页伪造用户消息，也不承诺从已经结束的对话中主动唤醒 Agent；此类唤醒由宿主负责。
 
 人工暂停由原生层锁定，模型不能通过 `set_control(true)` 恢复；只有当前授权面板的恢复按钮可以恢复。仅查看连接不能升级为控制连接，需重新连接并授权。关闭/隐藏面板仅停止预览，远程会话保留；断开按钮或 `crossdesk_disconnect` 才结束会话。
@@ -162,8 +164,16 @@ Apps 依赖宿主遵守 app-only 工具可见性和 `_meta` 的模型不可见
 
 动作包括 `move`、`click`、`double_click`、`scroll`、`drag`、`key_press`、`hotkey`、`type_text`。不提供 shell、任意键按下保持、剪贴板或文件操作。按键使用 Windows VK 对应的标准名称：`A`–`Z`、`0`–`9`、`F1`–`F12`、`CTRL`、`ALT`、`SHIFT`、`META`、`ENTER`、`ESC`、方向键等；应用快捷键仍需根据远端系统选择。
 
+鼠标动作和 `key_press` 支持可选的 `modifiers: ["CTRL", "ALT", "SHIFT", "META"]`（选择需要的键，不能重复），用于组合点击、选择、拖拽和快捷键。修饰键仅在本次动作期间保持，结束、取消或人工暂停时释放。`drag` 可指定 `button: "left" | "right" | "middle"`，默认左键。标点键支持 `SEMICOLON`、`EQUAL`、`COMMA`、`MINUS`、`PERIOD`、`SLASH`、`BACKQUOTE`、`BRACKET_LEFT`、`BACKSLASH`、`BRACKET_RIGHT`、`QUOTE`；这些是 VK 键位名称，实际字符取决于远端键盘布局，输入文字应使用 `type_text`。例如：
+
+```json
+{"session_id":"…","operation_id":"select-item-1","observation_id":"…","action":{"type":"click","x":120,"y":80,"modifiers":["SHIFT"]}}
+```
+
 截图默认最长边 1280，最大 2048；允许指定源帧像素区域。返回 `source_size`、`source_rect`、`image_size`、`generation`、`geometry_epoch`、`frame_id`、`frame_age_ms` 和 `observation_id`。动作坐标使用返回图像坐标，不使用浏览器 CSS 像素。观测保留最多 32 个，30 秒过期；切换显示器、连接代际或图像几何变化立即失效。截图使用不超过 2 秒的帧，PNG 不超过 2 MiB，必要时继续缩小输出并返回真实尺寸。
 
+`timeout_ms` 限制等待可用帧的时间，PNG 编码独立执行；`0` 表示只使用当前已就绪的新鲜帧，无帧返回 `FRAME_TIMEOUT`，编码器繁忙返回 `BUSY`。截图按到达顺序处理，并优先于预览。视频超过 2 秒未更新时拒绝新的键鼠输入，恢复新鲜画面后可继续；已提交操作的去重查询仍可读取原结果。
+
 `operation_id` 在当前原生会话中缓存最近 128 个操作，至少在未被容量淘汰的 5 分钟窗口内支持同参数查询；不同参数复用 ID 返回冲突。`submitted` 仅说明本地发送完成，远端没有执行 ACK。超时、断网、取消或发送失败可能返回 `outcome_unknown`/`side_effect_possible`，禁止自动重放。`observe_after` 默认开启，后续图像只证明取得更新帧，不证明操作执行成功。
 
 ## 实时视频与按需截图
@@ -211,3 +221,11 @@ npm run pack:npm -- --out-dir /absolute/path/artifacts
 脚本生成可发布 tarball，包含面板、运行代码、源代码、许可和固定版本的五个平台可选依赖。CI 的 `MCP package` 工作流仅构建并上传 `mcp-package`，不发布 npm。原生包继续由 CLI 工作流构建，`doctor` 必须同时声明自动化协议和 `automation_video_protocols: ["h264-annexb/1", "nv12/1"]` 和 `automation_video_decoders: ["auto", "software"]`，以及 `automation_video_transport: "loopback-websocket/1"`、`automation_terminal_protocol: "ansi/1"`；先发布对应 0.2.0 平台包，再发布 MCP tarball。尚未进行真实跨平台发布/远端验收时，不应把构建成功等同于各宿主和设备上的联调完成。
 
 从发布 tarball 重建 TypeScript/面板时，将包中的 `build-package.json` 复制为 `package.json`，运行 `npm ci && npm run build`。原生源码和依赖来源随平台包提供，详见 [源码来源](../cli/docs/source-provenance.md)。
+
+## 本地审阅验证（2026-11-03）
+
+动作参数校验和输入步骤生成集中在 `apps/cli/automation_protocol.*`，主循环负责调度、取消和权限检查。截图按色度网格对齐后仅转换所需区域，PNG 在容量受限的缓冲区中单次压缩；原生帧仅在没有其他读取者时复用内存。Node 按行累计响应字节，在完整响应到达时解码，避免反复扫描和复制大截图。
+
+本机 macOS arm64 完成 TypeScript/面板构建、原生 CLI 构建、现有 wire/keyboard-state 测试，以及工程外的 MCP、会话和原生动作回归检查。ASan/UBSan 检查覆盖动作调度和输入清理；140 组随机图像覆盖完整帧、奇偶裁剪边界、缩放及容量回退，优化前后 PNG 字节一致。
+
+使用 1920 × 1080 合成 NV12、最长边 1280、每项 6 次的 `-O2` 本地基准：完整画面编码约 22.5 → 11.5 ms，321 × 241 裁剪约 2.1 → 0.9 ms。数字仅衡量图像转换与编码，不包含网络、解码、远端执行或宿主渲染；未连接真实远端或完成 Windows/Linux 验收。额外测试代码、脚本、数据和产物均存放于工程外系统临时目录。

apps/mcp/src/browser.ts

@@ -367,6 +367,7 @@ export async function pairBrowser(
           resolve(undefined);
         });
         socket.once("error", () => resolve(undefined));
+        socket.once("close", () => resolve(undefined));
         socket.once("connect", () =>
           socket.write(
             JSON.stringify({ secret: descriptor.secret, view_id: view }) + "\n",

apps/mcp/src/common.ts

@@ -61,29 +61,38 @@ export const point = z.strictObject({
   y: z.number().finite().min(0).max(8192),
 });
 const button = z.enum(["left", "right", "middle"]).optional();
-const key = z
-  .string()
-  .min(1)
-  .max(16)
-  .regex(/^[A-Z0-9_]+$/);
+const unique = (keys: string[]) => new Set(keys).size === keys.length;
+const modifiers = z.array(z.enum(["CTRL", "ALT", "SHIFT", "META"])).max(4).refine(unique)
+  .describe("Modifiers held during this action and released afterwards, including on cancellation.").optional();
+const key = z.enum([
+  ..."ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789".split(""),
+  "BACKSPACE", "TAB", "ENTER", "SHIFT", "CTRL", "ALT", "PAUSE", "CAPS_LOCK",
+  "ESC", "SPACE", "PAGE_UP", "PAGE_DOWN", "END", "HOME", "LEFT", "UP", "RIGHT", "DOWN",
+  "INSERT", "DELETE", "META", "F1", "F2", "F3", "F4", "F5", "F6", "F7", "F8", "F9", "F10", "F11", "F12",
+  "SEMICOLON", "EQUAL", "COMMA", "MINUS", "PERIOD", "SLASH", "BACKQUOTE", "BRACKET_LEFT", "BACKSLASH", "BRACKET_RIGHT", "QUOTE",
+]);
 export const action = z.discriminatedUnion("type", [
-  point.extend({ type: z.literal("move") }),
-  point.extend({ type: z.literal("click"), button }),
-  point.extend({ type: z.literal("double_click"), button }),
+  point.extend({ type: z.literal("move"), modifiers }),
+  point.extend({ type: z.literal("click"), button, modifiers }),
+  point.extend({ type: z.literal("double_click"), button, modifiers }),
   point.extend({
     type: z.literal("scroll"),
     direction: z.enum(["up", "down", "left", "right"]),
     ticks: z.int().min(1).max(10).optional(),
+    modifiers,
   }),
   z.strictObject({
     type: z.literal("drag"),
     path: z.array(point).min(2).max(128),
     duration_ms: z.int().min(100).max(2000).optional(),
+    button,
+    modifiers,
   }),
-  z.strictObject({ type: z.literal("key_press"), key }),
+  z.strictObject({ type: z.literal("key_press"), key, modifiers })
+    .refine((p) => !p.modifiers?.some((modifier) => modifier === p.key)),
   z.strictObject({
     type: z.literal("hotkey"),
-    keys: z.array(key).min(2).max(6),
+    keys: z.array(key).min(2).max(6).refine(unique),
   }),
   z.strictObject({
     type: z.literal("type_text"),

apps/mcp/src/native.ts

@@ -95,8 +95,10 @@ export class NativeClient {
   private child: ChildProcess;
   private pending = new Map<number, Pending>();
   private sequence = 0;
-  private buffer = "";
+  private chunks: Buffer[] = [];
+  private bufferedBytes = 0;
   private closed = false;
+  private exited = false;
   private closing?: Promise<void>;
   constructor(binary: string, server: string, port: number, logs: string,
     readonly videoOptions: VideoOptions = { decode_path: "browser", decoder: "auto" }) {
@@ -122,12 +124,14 @@ export class NativeClient {
         env: nativeEnvironment(),
       },
     );
-    this.child.stdout!.setEncoding("utf8");
-    this.child.stdout!.on("data", (part: string) => this.receive(part));
+    this.child.stdout!.on("data", (part: Buffer) => this.receive(part));
     this.child.stdin!.on("error", () => this.fail());
     this.child.stdio[3]!.on("error", () => this.fail());
     this.child.once("error", () => this.fail());
-    this.child.once("exit", () => this.fail());
+    // Process exit can precede the final stdout data. Drain replies until EOF
+    // or close before rejecting requests that have no response.
+    this.child.once("close", () => { this.exited = true; this.fail(); });
+    this.child.stdout!.once("end", () => this.fail());
   }
   async connect(
     device: string,
@@ -161,8 +165,8 @@ export class NativeClient {
     timeout = 7000,
     signal?: AbortSignal,
   ): Promise<Data> {
-    if (this.closed)
-      return Promise.reject(new Failure("SESSION_CLOSED", method === "action"));
+    if (this.closed || (this.closing && method !== "disconnect"))
+      return Promise.reject(new Failure("SESSION_CLOSED"));
     if (signal?.aborted) return Promise.reject(new Failure("CANCELLED"));
     if (this.pending.size >= 32 || this.child.stdin!.writableLength > 262144)
       return Promise.reject(new Failure("BUSY", false, true));
@@ -201,22 +205,30 @@ export class NativeClient {
       this.child.stdin!.write(line);
     });
   }
-  private receive(part: string) {
-    this.buffer += part;
-    if (Buffer.byteLength(this.buffer) > 8 * 1024 * 1024) {
-      this.fail();
-      this.child.kill();
-      return;
-    }
-    for (;;) {
-      const end = this.buffer.indexOf("\n");
-      if (end < 0) break;
-      const line = this.buffer.slice(0, end);
-      this.buffer = this.buffer.slice(end + 1);
+  private receive(part: Buffer) {
+    if (this.closed) return;
+    for (let start = 0; start < part.length;) {
+      const end = part.indexOf(10, start);
+      const chunk = part.subarray(start, end < 0 ? part.length : end);
+      this.bufferedBytes += chunk.length;
+      if (this.bufferedBytes > 4 * 1024 * 1024) {
+        this.fail();
+        this.child.kill();
+        return;
+      }
+      this.chunks.push(chunk);
+      if (end < 0) return;
+      // One allocation and UTF-8 decode per response, including split UTF-8
+      // codepoints. Do not repeatedly scan/copy a multi-megabyte PNG response.
+      const line = this.chunks.length === 1 ? chunk : Buffer.concat(this.chunks, this.bufferedBytes);
+      this.chunks = [];
+      this.bufferedBytes = 0;
+      start = end + 1;
       try {
-        if (Buffer.byteLength(line) > 4 * 1024 * 1024) throw new Error();
-        const reply = JSON.parse(line);
-        if (!Number.isSafeInteger(reply.id) || typeof reply.ok !== "boolean")
+        const reply = JSON.parse(line.toString("utf8"));
+        if (!reply || !Number.isSafeInteger(reply.id) || reply.id < 0 || typeof reply.ok !== "boolean" ||
+            (reply.ok && (!reply.result || typeof reply.result !== "object" || Array.isArray(reply.result))) ||
+            (!reply.ok && (!reply.error || typeof reply.error.code !== "string")))
           throw new Error();
         const p = this.pending.get(reply.id);
         if (!p) continue;
@@ -242,6 +254,8 @@ export class NativeClient {
   }
   private fail() {
     this.closed = true;
+    this.chunks = [];
+    this.bufferedBytes = 0;
     for (const p of this.pending.values()) {
       p.clean();
       p.reject(new Failure("SESSION_CLOSED", p.action));
@@ -254,13 +268,13 @@ export class NativeClient {
       if (!this.closed)
         await this.request("disconnect", {}, 1000).catch(() => {});
       this.child.stdin?.end();
-      if (this.child.exitCode === null && this.child.signalCode === null)
+      if (!this.exited && this.child.exitCode === null && this.child.signalCode === null)
         await new Promise<void>((resolve) => {
           const kill = setTimeout(() => {
             this.child.kill("SIGKILL");
-            resolve();
           }, 2000);
-          this.child.once("exit", () => {
+          // Keep the device lock until the child really exits.
+          this.child.once("close", () => {
             clearTimeout(kill);
             resolve();
           });

apps/mcp/src/server.ts

@@ -203,17 +203,28 @@ export function createServer(
     "Connect only a saved, previously authorized device with available credentials. AUTH_REQUIRED means the user must authorize through the panel or setup.",
     async (p, ctx) => {
       const selected = sessions.choose(supported(ctx));
-      const result = await sessions.connectSaved(
-        p.device_alias,
-        p.mode,
-        p.display_index,
-        ctx.mcpReq.signal,
-      );
+      // Reserve panel capacity before starting a connection. Once connected,
+      // panel setup errors must not hide the live session from the caller.
       const v = sessions.create(selected, p.device_alias);
+      let result: Data;
+      try {
+        result = await sessions.connectSaved(
+          p.device_alias,
+          p.mode,
+          p.display_index,
+          ctx.mcpReq.signal,
+        );
+      } catch (error) {
+        sessions.views.delete(v.id);
+        throw error;
+      }
       v.session = result.session_id;
+      let panelResult: Data;
+      try { panelResult = await openPanel(v); }
+      catch (error) { panelResult = { ...sessions.describe(v), panel_error: failure(error) }; }
       return viewResult(v, {
         ...result,
-        ...await openPanel(v),
+        ...panelResult,
         state: result.state,
       });
     },

apps/mcp/src/sessions.ts

@@ -2,6 +2,7 @@ import { join } from "node:path";
 import { setTimeout as delay } from "node:timers/promises";
 import {
   Failure,
+  failure,
   token,
   equal,
   parse,
@@ -25,6 +26,7 @@ export interface View {
   alias?: string;
   session?: string;
   disconnected?: boolean;
+  connectionError?: Data;
 }
 interface Session {
   id: string;
@@ -32,11 +34,15 @@ interface Session {
   native: NativeClient;
   status: Data;
   release: () => Promise<void>;
+  connecting: boolean;
+  statusRequest?: Promise<Data>;
+  closing?: Promise<void>;
 }
 export class Sessions {
   readonly views = new Map<string, View>();
   private current?: Session;
-  private connecting = false;
+  private connecting?: AbortController;
+  private connectionDone?: Promise<void>;
   private closing = false;
   private binary?: Promise<string>;
   constructor(
@@ -94,11 +100,12 @@ export class Sessions {
           ? "not_confirmed"
           : "waiting",
       session_id: v.session ?? null,
-      state: v.session ? "session_available" : v.disconnected ? "closed" : "awaiting_user",
+      state: v.session ? "session_available" : v.disconnected ? "closed" : v.connectionError ? "failed" : "awaiting_user",
+      ...(v.connectionError ? { error: v.connectionError } : {}),
       device_alias: v.alias ?? null,
       next_step: v.session
         ? "Call crossdesk_get_session with view_id."
-        : v.disconnected
+        : v.disconnected || v.connectionError
           ? "The session was disconnected. Do not automatically reconnect."
           : v.panel === "off"
             ? "Use setup to configure credentials and authorization, then crossdesk_connect with the saved device alias. Never send passwords in chat."
@@ -117,25 +124,31 @@ export class Sessions {
     const s = this.session(id!);
     // A panel connection may still be performing the native hello handshake.
     // Reading native status here would race initialization and could close it.
-    if (this.connecting) return {
+    if (s.connecting) return {
       ...(view ? this.describe(view) : {}),
       session_id: s.id,
       device_alias: s.device.alias,
       state: "connecting",
       control: false,
     };
-    try {
-      s.status = await s.native.request("status", {}, 5000, signal);
-    } catch {
-      if (signal?.aborted) throw new Failure("CANCELLED");
-      s.status = { state: "closed", control: false };
+    // Concurrent panel/model polls share one native read. A cancelled waiter
+    // must not cancel another caller's status request or disconnect the session.
+    if (signal?.aborted) throw new Failure("CANCELLED");
+    if (!["closed", "failed"].includes(s.status.state)) {
+      s.statusRequest ??= s.native.request("status", {}, 5000)
+        .then((status) => (s.status = status))
+        .catch((error) => {
+          if (!(error instanceof Failure) || error.code !== "SESSION_CLOSED") throw error;
+          return (s.status = { state: "closed", control: false });
+        })
+        .finally(() => { s.statusRequest = undefined; });
+      await this.waitStatus(s.statusRequest, signal);
     }
     // The user may have disconnected or replaced the session during the read.
     if (this.current !== s || (view && view.session !== id))
       return view ? this.describe(view) : { session_id: id, state: "closed", control: false };
     if (["closed", "failed"].includes(s.status.state)) {
-      await s.native.close();
-      await s.release();
+      await this.dispose(s);
     }
     return {
       ...s.status,
@@ -150,6 +163,22 @@ export class Sessions {
       } : {}),
     };
   }
+  private async waitStatus(request: Promise<Data>, signal?: AbortSignal) {
+    if (!signal) return request;
+    let abort: () => void = () => {};
+    try {
+      return await Promise.race([request, new Promise<never>((_, reject) => {
+        abort = () => reject(new Failure("CANCELLED"));
+        signal.addEventListener("abort", abort, { once: true });
+        if (signal.aborted) abort();
+      })]);
+    } finally {
+      signal.removeEventListener("abort", abort);
+    }
+  }
+  private dispose(s: Session): Promise<void> {
+    return s.closing ??= s.native.close().finally(() => s.release());
+  }
   async waitForConnection(viewId: string, waitMs: number, signal?: AbortSignal): Promise<Data> {
     const deadline = Date.now() + waitMs;
     let result = this.describe(this.view(viewId));
@@ -253,18 +282,23 @@ export class Sessions {
   ): Promise<Data> {
     if (this.closing) throw new Failure("SESSION_CLOSED");
     if (this.connecting) throw new Failure("BUSY");
-    if (
-      this.current &&
-      !["closed", "failed", "disconnected"].includes(
-        (await this.status(this.current.id)).state,
-      )
-    )
-      throw new Failure("SESSION_LIMIT");
-    this.connecting = true;
+    // Reserve the single connection slot before the first await.
+    const controller = this.connecting = new AbortController();
+    signal = signal ? AbortSignal.any([signal, controller.signal]) : controller.signal;
+    let complete!: () => void;
+    this.connectionDone = new Promise<void>((resolve) => { complete = resolve; });
     let release: (() => Promise<void>) | undefined;
     let native: NativeClient | undefined;
+    let session: Session | undefined;
     try {
+      if (this.current && !["closed", "failed", "disconnected"].includes(
+        (await this.status(this.current.id, undefined, signal)).state,
+      )) throw new Failure("SESSION_LIMIT");
       if (this.current) await this.disconnect(this.current.id);
+      if (view) {
+        view.disconnected = false;
+        view.connectionError = undefined;
+      }
       release = await this.store.lock(
         `device:${d.server}:${d.port}:${d.device_id}`,
       );
@@ -281,12 +315,13 @@ export class Sessions {
         join(this.store.directory, "logs"),
         this.videoOptions ?? (await this.store.config()).video,
       );
-      const s: Session = {
+      const s: Session = session = {
         id: token(),
         device: d,
         native,
         status: {},
         release,
+        connecting: true,
       };
       this.current = s;
       if (view) {
@@ -306,8 +341,11 @@ export class Sessions {
       while (Date.now() < deadline) {
         if (this.closing || signal?.aborted) throw new Failure("CANCELLED");
         s.status = await native.request("status", {}, 5000, signal);
-        if (s.status.state === "ready" && s.status.frame_age_ms <= 2000)
+        if (this.current !== s || signal.aborted) throw new Failure("CANCELLED");
+        if (s.status.state === "ready" && typeof s.status.frame_age_ms === "number" && s.status.frame_age_ms <= 2000) {
+          s.connecting = false;
           return { ...s.status, session_id: s.id, device_alias: s.device.alias };
+        }
         if (["failed", "closed", "disconnected"].includes(s.status.state))
           throw new Failure(
             s.status.error === "incorrect_password"
@@ -318,14 +356,23 @@ export class Sessions {
       }
       throw new Failure("CONNECTION_TIMEOUT");
     } catch (error) {
-      await native?.close();
-      await release?.();
+      if (session) await this.dispose(session);
+      else {
+        await native?.close();
+        await release?.();
+      }
       if (this.current?.native === native) this.current = undefined;
-      if (view) view.session = undefined;
-      throw error;
+      const reason = signal.aborted ? new Failure("CANCELLED") : error;
+      if (view && (!view.session || view.session === session?.id)) {
+        view.session = undefined;
+        if (!view.disconnected) view.connectionError = failure(reason);
+      }
+      throw reason;
     } finally {
       password = "";
-      this.connecting = false;
+      this.connecting = undefined;
+      this.connectionDone = undefined;
+      complete();
     }
   }
   async screenshot(p: Data, signal?: AbortSignal): Promise<Data> {
@@ -369,14 +416,15 @@ export class Sessions {
   }
   async disconnect(id: string) {
     const s = this.session(id);
-    await s.native.close();
-    await s.release();
-    if (this.current === s) this.current = undefined;
+    if (s.connecting) this.connecting?.abort();
     for (const v of this.views.values())
       if (v.session === id) {
         v.session = undefined;
         v.disconnected = true;
+        v.connectionError = undefined;
       }
+    await this.dispose(s);
+    if (this.current === s) this.current = undefined;
     return { session_id: id, state: "closed" };
   }
   async viewerControl(v: View, operation: string) {
@@ -428,7 +476,10 @@ export class Sessions {
   }
   async close() {
     this.closing = true;
+    this.connecting?.abort();
+    const pending = this.connectionDone;
     if (this.current) await this.disconnect(this.current.id);
+    await pending;
     this.views.clear();
   }
 }

apps/mcp/src/ui-actions.ts

@@ -35,7 +35,7 @@ export async function uiAction(
       };
     }
     case "status":
-      return sessions.status(undefined, view.id);
+      return sessions.status(undefined, view.id, signal);
     case "connect":
       return sessions.connectUI(view, p, signal);
     case "control":