mirror of
https://github.com/kunkundi/crossdesk.git
synced 2026-11-03 04:22:43 +08:00
Compare commits
5
Commits
cli-v0.1.5
...
mcp
| Author | SHA1 | Date | |
|---|---|---|---|
 dijunkun
|
f57b4cc4e3 | ||
|
dijunkun
|
2fab61871e | ||
|
dijunkun
|
d776889562 | ||
|
dijunkun
|
30d409ff49 | ||
|
dijunkun
|
6de7165fb0 |
@@ -0,0 +1,34 @@
|
||||
name: MCP package
|
||||
|
||||
on:
|
||||
push:
|
||||
paths: ['apps/mcp/**', 'apps/cli/package.json', 'apps/cli/npm/platforms.json', '.github/workflows/build-mcp.yml']
|
||||
pull_request:
|
||||
paths: ['apps/mcp/**', 'apps/cli/package.json', 'apps/cli/npm/platforms.json', '.github/workflows/build-mcp.yml']
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
package:
|
||||
runs-on: ubuntu-22.04
|
||||
timeout-minutes: 15
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
- uses: actions/setup-node@v6
|
||||
with:
|
||||
node-version: 22
|
||||
cache: npm
|
||||
cache-dependency-path: apps/mcp/package-lock.json
|
||||
- run: npm ci
|
||||
working-directory: apps/mcp
|
||||
- run: npm run build
|
||||
working-directory: apps/mcp
|
||||
- run: npm run pack:npm
|
||||
working-directory: apps/mcp
|
||||
- uses: actions/upload-artifact@v6
|
||||
with:
|
||||
name: mcp-package
|
||||
path: build/npm-mcp/*.tgz
|
||||
if-no-files-found: error
|
||||
@@ -22,7 +22,7 @@
|
||||
|
||||
</div>
|
||||
|
||||
CrossDesk 让你从电脑、浏览器、手机或终端连接远端电脑，也能通过 **桌面墙（DeskHub）** 集中查看与操控多台设备。项目基于 [MiniRTC](https://github.com/kunkundi/minirtc)，支持自托管。
|
||||
CrossDesk 让你从电脑、浏览器、手机或终端（桌面 CLI）连接远端电脑，也能通过桌面墙（DeskHub）集中查看与操控多台设备。项目基于 [MiniRTC](https://github.com/kunkundi/minirtc)，支持自托管。
|
||||
|
||||
[主要功能](#features) · [下载与平台支持](#download) · [快速连接](#quick-start) · [终端 CLI](#cli) · [IP 直连](#lan-direct) · [可信设备](#trusted-devices) · [界面预览](#preview) · [使用指南](docs/USER_GUIDE.md) · [自托管](#self-hosting) · [构建与反馈](#development) · [赞助](#sponsor)
|
||||
|
||||
@@ -33,7 +33,7 @@ CrossDesk 让你从电脑、浏览器、手机或终端连接远端电脑，也
|
||||
- **跨平台远控：** Windows、macOS、Linux 桌面互控，浏览器及原生 iOS / Android 客户端访问电脑；鸿蒙 PC 与移动端处于开发阶段。
|
||||
- **局域网 IP 直连：** 桌面客户端通过 IP、端口和六位密码连接，无需云端信令服务；支持独立端口、记住密码及按模式显示近期设备。
|
||||
- **终端主控：** 在终端中通过设备 ID 观看和操控远程桌面，支持 ANSI / Kitty / Sixel 画面、局部缩放与平移、持续本地命令模式及 Pi 扩展，可单独构建。
|
||||
- **桌面墙：** 多设备实时预览、分组、放大操控、群控和批量文件分发。
|
||||
- 桌面墙：多设备实时预览、分组、放大操控、群控和批量文件分发。
|
||||
- **画面与声音：** H.264 / AV1、硬件编解码、远端音频，以及会话内画质和帧率调整；可用能力随平台与设备而异。
|
||||
- **日常协作：** 多显示器切换、键鼠与组合键、文本剪贴板、文件传输。
|
||||
- **可信设备：** 桌面端可选开启可信设备限制，在连接密码之外校验设备身份；支持复制、粘贴配对信息，提前添加允许访问本机的设备。
|
||||
@@ -157,7 +157,7 @@ CrossDesk 让你从电脑、浏览器、手机或终端连接远端电脑，也
|
||||
| ANSI 字符画面 | Kitty 协议像素画面 |
|
||||
| --- | --- |
|
||||
| <img src="docs/images/cli-ansi-overview.png" width="480" alt="CrossDesk CLI 的 ANSI 四象限全景：顶部显示只读模式和连接状态，底部显示操作快捷键" /> | <img src="docs/images/cli-kitty-overview.png" width="480" alt="CrossDesk CLI 的 Kitty 协议全景：同一远程桌面以像素画面显示，保留状态栏和操作快捷键" /> |
|
||||
| 用彩色字符显示桌面轮廓，适合普通终端；查看局部可按 `d` 放大。 | 直接显示像素画面，适合支持 Kitty 图形协议的终端，如 Ghostty。 |
|
||||
| 用彩色字符显示桌面轮廓，适合普通终端；查看局部可用 `+/-` 或滚轮以鼠标位置缩放。 | 直接显示像素画面，适合支持 Kitty 图形协议的终端，如 Ghostty。 |
|
||||
|
||||
两图采集于 **2026-10-31**，来自同一授权设备的实际只读连接，以 128 × 40 字符视口还原 CLI 的 ANSI / Kitty 协议输出。图中 FPS 是采集时的实际输出计数，静止画面下 ANSI 无变化可显示 `FPS 0`。图示为输出渲染，终端实窗效果受字体、窗口大小及协议支持影响；采集方法见 [图示来源](docs/images/README.md#cli-captures)。
|
||||
|
||||
@@ -227,7 +227,7 @@ crossdesk-cli 123456789 --quality balanced --fps 8
|
||||
| `--control` | 连接后直接启用键鼠控制；省略时默认只读 |
|
||||
| `--server your-server.example --port 9099` | 使用自建信令服务，两端需接入同一服务 |
|
||||
|
||||
ANSI 全景受字符网格限制，提高画质不能让整屏小字变得清晰；可扩大终端、缩小字体，再按 `d` 查看局部，按 `f` 回到全景。顶部状态栏显示模式、连接进度、渲染器、缩放、实时输出 FPS 和网络信息，窄窗口会省略部分字段。通过 SSH 使用时分配 PTY（`ssh -t`）；tmux / screen 使用 ANSI。更多显示、鼠标报告和兼容性说明见 [终端使用指南](apps/cli/docs/terminal-client.md)。
|
||||
ANSI 全景受字符网格限制，提高画质不能让整屏小字变得清晰；可扩大终端、缩小字体，再用 `+/-` 或滚轮查看局部，按 `f` 回到全景并重置平移。顶部状态栏显示模式、连接进度、渲染器、缩放、实时输出 FPS 和网络信息，窄窗口会省略部分字段。通过 SSH 使用时分配 PTY（`ssh -t`）；tmux / screen 使用 ANSI。更多显示、鼠标报告和兼容性说明见 [终端使用指南](apps/cli/docs/terminal-client.md)。
|
||||
|
||||
### 控制、查看与退出
|
||||
|
||||
@@ -237,10 +237,13 @@ ANSI 全景受字符网格限制，提高画质不能让整屏小字变得清晰
|
||||
| 控制模式 | 键鼠输入传给远端；`Ctrl+]` 进入本地命令模式并暂停远端输入 |
|
||||
| 本地命令模式 | 执行命令后持续保持；`c` 选择手动返回后是否启用控制，`q` 退出 CLI |
|
||||
| `Esc` / `Ctrl+C` | 在本地命令模式中手动返回所选的控制或只读模式 |
|
||||
| `+` / `=` / `-` | 在只读或本地命令模式中放大 / 缩小 |
|
||||
| `d` / `f` | 在只读或本地命令模式中切换原始细节 / 完整桌面；只读时也可点击目标进入细节 |
|
||||
| `+` / `=` / `-` | 在只读或本地命令模式中以鼠标位置放大 / 缩小 |
|
||||
| 滚轮 / 点击 | 只读模式中，滚轮以鼠标位置缩放；点击目标进入原始细节 |
|
||||
| `f` | 在只读或本地命令模式中恢复完整桌面，同时重置倍率、平移和细节模式 |
|
||||
| 方向键 / `i j k l` | 在只读或本地命令模式中平移；`i` 上、`j` 左、`k` 下、`l` 右 |
|
||||
|
||||
鼠标下的画面内容在缩放时尽量保持原位，到达桌面边缘时限制裁剪范围；没有有效鼠标位置时沿用当前中心。本地命令模式中鼠标仅更新缩放锚点，点击和滚轮不执行视图操作。
|
||||
|
||||
**控制模式下退出：先按 `Ctrl+]`，再按 `q`。** 普通 `c`、`q` 和 `Ctrl+C` 在控制模式中会发送到远端。
|
||||
|
||||
`Ctrl+]` 模式不会因为缩放、平移或按 `c` 自动结束。状态栏中的 `CONTROL PAUSED` / `READ ONLY` 表示返回后是否启用控制；按 `Esc` / `Ctrl+C` 手动返回。再次按 `Ctrl+]` 也可返回，若返回控制模式则向远端发送一次该组合键。
|
||||
|
||||
+9
-6
@@ -22,7 +22,7 @@
|
||||
|
||||
</div>
|
||||
|
||||
CrossDesk connects your computers from desktop, browser, mobile, and terminal clients, with **DeskHub** for viewing and controlling multiple devices together. It is built on [MiniRTC](https://github.com/kunkundi/minirtc) and supports self-hosting.
|
||||
CrossDesk connects your computers from desktop, browser, mobile, and terminal clients, with DeskHub for viewing and controlling multiple devices together. It is built on [MiniRTC](https://github.com/kunkundi/minirtc) and supports self-hosting.
|
||||
|
||||
[Features](#features) · [Download and platforms](#download) · [Quick start](#quick-start) · [Terminal CLI](#cli) · [Direct IP](#lan-direct) · [Trusted devices](#trusted-devices) · [Preview](#preview) · [User guide](docs/USER_GUIDE_EN.md) · [Self-hosting](#self-hosting) · [Build and feedback](#development) · [Sponsor](#sponsor)
|
||||
|
||||
@@ -33,7 +33,7 @@ CrossDesk connects your computers from desktop, browser, mobile, and terminal cl
|
||||
- **Cross-platform access:** control Windows, macOS, and Linux desktops from computers, browsers, or native iOS / Android clients. HarmonyOS PC and mobile clients are in development.
|
||||
- **Direct IP over LAN:** connect desktop clients using an IP address, port, and six-character password without a cloud signaling service. Each local IP has its own port; passwords can be remembered and recent devices are separated by mode.
|
||||
- **Terminal controller:** view and control remote desktops by device ID, with ANSI / Kitty / Sixel rendering, zoom and pan, persistent local command mode, a Pi extension, and a build independent of the desktop GUI.
|
||||
- **DeskHub:** live multi-device previews, groups, enlarged control, group input, and batch file distribution.
|
||||
- DeskHub: live multi-device previews, groups, enlarged control, group input, and batch file distribution.
|
||||
- **Video and audio:** H.264 / AV1, hardware codecs, remote audio, and in-session quality and frame-rate controls. Capabilities depend on the platform and device.
|
||||
- **Everyday controls:** monitor switching, keyboard and mouse input, shortcuts, text clipboard, and file transfer.
|
||||
- **Trusted devices:** optionally restrict desktop access to trusted device identities in addition to the connection password. Copy and paste pairing information to approve devices before they connect.
|
||||
@@ -157,7 +157,7 @@ Remote sessions support monitor switching, video quality and frame-rate controls
|
||||
| ANSI character rendering | Kitty pixel graphics |
|
||||
| --- | --- |
|
||||
| <img src="docs/images/cli-ansi-overview.png" width="480" alt="CrossDesk CLI ANSI quadrant overview, with read-only mode and connection status at the top and shortcuts at the bottom" /> | <img src="docs/images/cli-kitty-overview.png" width="480" alt="CrossDesk CLI Kitty protocol overview of the same remote desktop, rendered as pixels with the status bar and shortcuts" /> |
|
||||
| Colored characters show the desktop in standard terminals; press `d` to inspect a smaller region. | Pixel graphics display the desktop in terminals supporting the Kitty graphics protocol, such as Ghostty. |
|
||||
| Colored characters show the desktop in standard terminals; use `+/-` or the wheel to zoom at the mouse position. | Pixel graphics display the desktop in terminals supporting the Kitty graphics protocol, such as Ghostty. |
|
||||
|
||||
Both images were captured on **2026-10-31** from real read-only connections to the same authorized host. They render the CLI's ANSI / Kitty protocol output in a 128 × 40 character viewport. FPS counts actual output during capture; an unchanged ANSI view can show `FPS 0`. Actual terminal appearance depends on the font, window size, and protocol support. See [capture notes](docs/images/README.md#cli-captures) for the method.
|
||||
|
||||
@@ -227,7 +227,7 @@ crossdesk-cli 123456789 --quality balanced --fps 8
|
||||
| `--control` | Enable keyboard/mouse control immediately after connecting; omitted by default for read-only viewing |
|
||||
| `--server your-server.example --port 9099` | Use a self-hosted signaling service; both ends must use the same service |
|
||||
|
||||
ANSI overviews are limited by the character grid: higher video quality alone cannot make full-desktop text legible. Enlarge the terminal or reduce its font size, press `d` for detail, and `f` to return to the full desktop. The top bar shows input mode, connection progress, renderer, zoom, live output FPS, and network information; narrow windows omit some fields. Allocate a PTY over SSH (`ssh -t`), and use ANSI inside tmux / screen. See the [terminal guide](apps/cli/docs/terminal-client.md) (Chinese) for rendering, mouse reporting, and compatibility details.
|
||||
ANSI overviews are limited by the character grid: higher video quality alone cannot make full-desktop text legible. Enlarge the terminal or reduce its font size, use `+/-` or the wheel for detail, and `f` to restore the full desktop and reset panning. The top bar shows input mode, connection progress, renderer, zoom, live output FPS, and network information; narrow windows omit some fields. Allocate a PTY over SSH (`ssh -t`), and use ANSI inside tmux / screen. See the [terminal guide](apps/cli/docs/terminal-client.md) (Chinese) for rendering, mouse reporting, and compatibility details.
|
||||
|
||||
### Control, navigation, and exit
|
||||
|
||||
@@ -237,10 +237,13 @@ ANSI overviews are limited by the character grid: higher video quality alone can
|
||||
| Control mode | Keyboard/mouse input goes to the host; `Ctrl+]` enters local command mode and pauses remote input |
|
||||
| Local command mode | Stays active after commands; `c` selects whether control resumes when you leave; `q` exits the CLI |
|
||||
| `Esc` / `Ctrl+C` | Manually leave local command mode and return to the selected control or read-only mode |
|
||||
| `+` / `=` / `-` | Zoom in / out in read-only or local command mode |
|
||||
| `d` / `f` | Switch to native detail / full-desktop view in read-only or local command mode; clicking in read-only mode also opens detail at that point |
|
||||
| `+` / `=` / `-` | Zoom in / out at the mouse position in read-only or local command mode |
|
||||
| Wheel / click | In read-only mode, zoom at the mouse position with the wheel; click a point to open native detail |
|
||||
| `f` | Restore the full desktop and reset zoom, pan, and native detail mode in read-only or local command mode |
|
||||
| Arrow keys / `i j k l` | Pan in read-only or local command mode: `i` up, `j` left, `k` down, `l` right |
|
||||
|
||||
Zoom keeps the content under the pointer in place where possible, clamping the crop at desktop edges. Without a valid mouse position, zoom uses the current center. In local command mode, mouse reports only update the zoom anchor; clicks and the wheel do not change the view.
|
||||
|
||||
**To quit while controlling the host, press `Ctrl+]`, then `q`.** Ordinary `c`, `q`, and `Ctrl+C` go to the host in control mode.
|
||||
|
||||
Zooming, panning, or pressing `c` does not end `Ctrl+]` mode. The status bar shows `CONTROL PAUSED` / `READ ONLY` to indicate whether control will resume; press `Esc` / `Ctrl+C` to return manually. Pressing `Ctrl+]` again also leaves local mode and forwards that shortcut if control resumes.
|
||||
|
||||
+11
-5
@@ -59,23 +59,29 @@ crossdesk-cli 123456789 --quality balanced --fps 8
|
||||
crossdesk-cli 123456789 --display 1
|
||||
```
|
||||
|
||||
默认请求 `high` 画质（最高 1920 × 1080），`--display` 的索引从 `0` 开始。交互 CLI 默认本地输出目标为 30 FPS，远端供帧会随输出耗时自适应；显式 `--fps` 固定目标和相应预览请求，实际帧率仍取决于设备、终端与链路。更多参数见 [主 README](../../README.md#cli) 和 [终端使用指南](docs/terminal-client.md)。
|
||||
默认请求 `high` 画质（最高 1920 × 1080）和清晰度优先策略，`--display` 的索引从 `0` 开始。交互 CLI 默认本地输出目标为 30 FPS，远端供帧会随输出耗时自适应；显式 `--fps` 固定目标和相应预览请求，各帧率下均优先保持分辨率，实际画质与帧率仍取决于设备、终端与链路。更多参数见 [主 README](../../README.md#cli) 和 [终端使用指南](docs/terminal-client.md)。
|
||||
|
||||
## 查看、控制与退出
|
||||
|
||||
| 操作 | 按键 |
|
||||
| --- | --- |
|
||||
| 开启远端控制 | 只读模式下按 `c` |
|
||||
| 放大局部 / 回到全景 | 只读或本地命令模式下按 `d` / `f`；只读时也可点击目标进入细节 |
|
||||
| 缩放 / 平移 | 只读或本地命令模式下用 `+/-` 或滚轮缩放；方向键或 `i` 上、`j` 左、`k` 下、`l` 右平移 |
|
||||
| 缩放 | 只读或本地命令模式下用 `+` / `=` / `-` 以鼠标位置缩放；只读时也可用滚轮 |
|
||||
| 回到全景 | 只读或本地命令模式下按 `f`，同时重置倍率、平移和细节模式 |
|
||||
| 平移 | 只读或本地命令模式下用方向键或 `i` 上、`j` 左、`k` 下、`l` 右 |
|
||||
| 原始细节 | 只读时点击目标位置，或以 `--zoom native` 启动 |
|
||||
| 暂停远端输入并进入本地命令模式 | `Ctrl+]` |
|
||||
| 选择返回后的控制状态 | 本地命令模式下按 `c`，再按 `Esc` / `Ctrl+C` 返回 |
|
||||
| 退出连接 | 只读模式下按 `q` / `Ctrl+C`；控制模式下先按 `Ctrl+]`，再按 `q` |
|
||||
|
||||
本地命令模式在缩放、平移和按 `c` 后持续保持，直到手动返回。控制模式中的 `Ctrl+T`、`Ctrl+C` 和普通按键均传给远端。再次按 `Ctrl+]` 也可离开本地命令模式，若返回控制模式则同时向远端发送该组合键。
|
||||
缩放尽量保持鼠标下的画面内容原位；没有有效鼠标位置时沿用当前中心。底部提示随模式和窗口宽度调整，`Zoom at mouse` 表示鼠标锚点缩放，`Full view` / `Full` 表示恢复全景。
|
||||
|
||||
本地命令模式在缩放、平移和按 `c` 后持续保持，直到手动返回。此模式中鼠标仅更新缩放锚点，点击和滚轮不执行视图操作。控制模式中的 `Ctrl+T`、`Ctrl+C` 和普通按键均传给远端。再次按 `Ctrl+]` 也可离开本地命令模式，若返回控制模式则同时向远端发送该组合键。
|
||||
|
||||
## 更多文档与功能范围
|
||||
|
||||
- [MCP 服务与实时面板](../mcp/README.md)：Agent 截图和结构化远程操作，不要求控制端安装桌面客户端。
|
||||
- [原生自动化协议](docs/automation-protocol.md)：独立 `agent --stdio` 模式。
|
||||
- [构建、使用、Pi 安装和故障排查](docs/terminal-client.md)
|
||||
- [私有桥接协议](docs/bridge-protocol.md)
|
||||
- [实际验证记录](docs/validation.md)
|
||||
@@ -83,4 +89,4 @@ crossdesk-cli 123456789 --display 1
|
||||
|
||||
CLI 仅作为控制端，每次连接通过服务端获得新的 `cli-xxxxxxxxx` 临时 ID。使用前须部署支持 `cli` 注册的 crossdesk-server；旧服务端不会回退为普通设备身份。
|
||||
|
||||
终端 I/O 包含 POSIX（Linux/macOS）和 Windows Console / ConPTY 实现。macOS arm64 已完整构建，之前已连接真实被控端验证 ANSI 观看与预览协商。Linux / Windows 全量构建和真实连接、SSH/Kitty/Sixel 实机显示及真实 Pi TUI 仍需验收；交叉编译检查不等于完整运行验证。此项目不创建被控端虚拟桌面，不提供音频播放、文件传输、剪贴板同步或 MCP 服务。
|
||||
终端 I/O 包含 POSIX（Linux/macOS）和 Windows Console / ConPTY 实现。macOS arm64 已完整构建，之前已连接真实被控端验证 ANSI 观看与预览协商。Linux / Windows 全量构建和真实连接、SSH/Kitty/Sixel 实机显示及真实 Pi TUI 仍需验收；交叉编译检查不等于完整运行验证。CLI 不创建被控端虚拟桌面，不提供音频播放、文件传输或剪贴板同步；MCP 服务由独立的 `apps/mcp` 包提供。
|
||||
|
||||
@@ -0,0 +1,894 @@
|
||||
#include "agent.h"
|
||||
#include "terminal_io.h"
|
||||
#include "video_server.h"
|
||||
#include <algorithm>
|
||||
#include <cerrno>
|
||||
#include <cmath>
|
||||
#include <deque>
|
||||
#include <frame_image.h>
|
||||
#include <future>
|
||||
#include <map>
|
||||
#include <memory>
|
||||
#include <stdexcept>
|
||||
|
||||
namespace crossdesk::cli {
|
||||
namespace {
|
||||
using nlohmann::json;
|
||||
using Clock = std::chrono::steady_clock;
|
||||
using Ms = std::chrono::milliseconds;
|
||||
constexpr size_t kMaxResponse = 4 * 1024 * 1024;
|
||||
void Fields(const json &j, std::initializer_list<const char *> names) {
|
||||
if (!j.is_object())
|
||||
throw std::invalid_argument("INVALID_REQUEST");
|
||||
for (auto it = j.begin(); it != j.end(); ++it)
|
||||
if (std::none_of(names.begin(), names.end(),
|
||||
[&](const char *s) { return it.key() == s; }))
|
||||
throw std::invalid_argument("UNKNOWN_FIELD");
|
||||
}
|
||||
int Number(const json &j, const char *key, int low, int high, int fallback) {
|
||||
if (!j.contains(key))
|
||||
return fallback;
|
||||
if (!j[key].is_number_integer())
|
||||
throw std::invalid_argument("INVALID_REQUEST");
|
||||
const auto n = j[key].get<int64_t>();
|
||||
if (n < low || n > high)
|
||||
throw std::invalid_argument("INVALID_REQUEST");
|
||||
return int(n);
|
||||
}
|
||||
uint64_t Sequence(const json &j, const char *key, uint64_t fallback = 0) {
|
||||
if (!j.contains(key))
|
||||
return fallback;
|
||||
if (!j[key].is_number_unsigned())
|
||||
throw std::invalid_argument("INVALID_REQUEST");
|
||||
return j[key].get<uint64_t>();
|
||||
}
|
||||
bool Flag(const json &j, const char *key, bool fallback = false) {
|
||||
if (!j.contains(key))
|
||||
return fallback;
|
||||
if (!j[key].is_boolean())
|
||||
throw std::invalid_argument("INVALID_REQUEST");
|
||||
return j[key].get<bool>();
|
||||
}
|
||||
std::string String(const json &j, const char *key, size_t limit = 128) {
|
||||
if (!j.contains(key) || !j[key].is_string())
|
||||
throw std::invalid_argument("INVALID_REQUEST");
|
||||
auto s = j[key].get<std::string>();
|
||||
if (s.empty() || s.size() > limit || s.find('\0') != std::string::npos)
|
||||
throw std::invalid_argument("INVALID_REQUEST");
|
||||
return s;
|
||||
}
|
||||
int64_t Age(Clock::time_point queued) {
|
||||
return std::chrono::duration_cast<Ms>(Clock::now() - queued).count();
|
||||
}
|
||||
class Responses {
|
||||
public:
|
||||
explicit Responses(int fd) : fd_(fd), mode_(fd, true) {}
|
||||
void Put(json j) {
|
||||
auto line = j.dump(-1, ' ', false, json::error_handler_t::replace) + '\n';
|
||||
if (line.size() > kMaxResponse || queue_.size() >= 64 ||
|
||||
bytes_ + line.size() > 8 * 1024 * 1024)
|
||||
throw std::runtime_error("OUTPUT_BACKPRESSURE");
|
||||
if (queue_.empty())
|
||||
progressed_ = Clock::now();
|
||||
bytes_ += line.size();
|
||||
queue_.push_back(std::move(line));
|
||||
}
|
||||
bool Empty() const { return queue_.empty(); }
|
||||
void Pump() {
|
||||
if (Empty())
|
||||
return;
|
||||
const auto &s = queue_.front();
|
||||
auto n = Write(fd_, s.data() + offset_,
|
||||
std::min(size_t(16384), s.size() - offset_));
|
||||
if (n > 0) {
|
||||
offset_ += n;
|
||||
bytes_ -= n;
|
||||
progressed_ = Clock::now();
|
||||
if (offset_ == s.size()) {
|
||||
queue_.pop_front();
|
||||
offset_ = 0;
|
||||
}
|
||||
} else if (n < 0 && errno != EAGAIN && errno != EINTR)
|
||||
throw std::runtime_error("OUTPUT_CLOSED");
|
||||
if (Clock::now() - progressed_ > std::chrono::seconds(10))
|
||||
throw std::runtime_error("OUTPUT_TIMEOUT");
|
||||
}
|
||||
|
||||
private:
|
||||
int fd_;
|
||||
NonBlocking mode_;
|
||||
std::deque<std::string> queue_;
|
||||
size_t offset_ = 0, bytes_ = 0;
|
||||
Clock::time_point progressed_ = Clock::now();
|
||||
};
|
||||
struct Observation {
|
||||
std::string id;
|
||||
uint64_t generation, epoch;
|
||||
int width, height, source_width, source_height;
|
||||
ImageRegion region;
|
||||
Clock::time_point created;
|
||||
};
|
||||
struct Capture {
|
||||
json id, request;
|
||||
bool preview = false;
|
||||
Clock::time_point deadline;
|
||||
};
|
||||
struct Encoding {
|
||||
Capture capture;
|
||||
std::shared_ptr<const DecodedFrame> frame;
|
||||
uint64_t epoch;
|
||||
bool cancelled = false;
|
||||
std::future<FrameImage> result;
|
||||
};
|
||||
struct Step {
|
||||
int delay = 0, kind = 0, key = 0, flag = 0, wheel = 0;
|
||||
float x = 0, y = 0;
|
||||
bool down = false;
|
||||
std::string text;
|
||||
};
|
||||
struct RunningAction {
|
||||
json id;
|
||||
std::string operation;
|
||||
uint64_t generation = 0, epoch = 0;
|
||||
std::vector<Step> steps;
|
||||
size_t next = 0;
|
||||
bool submitted = false;
|
||||
Clock::time_point started;
|
||||
};
|
||||
struct Record {
|
||||
std::string operation, arguments;
|
||||
json result;
|
||||
Clock::time_point created;
|
||||
};
|
||||
int KeyCode(const std::string &key) {
|
||||
if (key.size() == 1 &&
|
||||
((key[0] >= 'A' && key[0] <= 'Z') || (key[0] >= '0' && key[0] <= '9')))
|
||||
return key[0];
|
||||
static const std::map<std::string, int> keys = {
|
||||
{"BACKSPACE", 8}, {"TAB", 9}, {"ENTER", 13}, {"SHIFT", 16},
|
||||
{"CTRL", 17}, {"ALT", 18}, {"PAUSE", 19}, {"CAPS_LOCK", 20},
|
||||
{"ESC", 27}, {"SPACE", 32}, {"PAGE_UP", 33}, {"PAGE_DOWN", 34},
|
||||
{"END", 35}, {"HOME", 36}, {"LEFT", 37}, {"UP", 38},
|
||||
{"RIGHT", 39}, {"DOWN", 40}, {"INSERT", 45}, {"DELETE", 46},
|
||||
{"META", 91}, {"F1", 112}, {"F2", 113}, {"F3", 114},
|
||||
{"F4", 115}, {"F5", 116}, {"F6", 117}, {"F7", 118},
|
||||
{"F8", 119}, {"F9", 120}, {"F10", 121}, {"F11", 122},
|
||||
{"F12", 123}};
|
||||
auto it = keys.find(key);
|
||||
if (it == keys.end())
|
||||
throw std::invalid_argument("INVALID_KEY");
|
||||
return it->second;
|
||||
}
|
||||
class Automation {
|
||||
public:
|
||||
Automation(int output, SessionOptions options, int password, bool video)
|
||||
: output_(output), options_(std::move(options)), password_fd_(password),
|
||||
video_(video ? std::make_unique<VideoServer>() : nullptr) {}
|
||||
~Automation() { session_.Disconnect(); }
|
||||
int Run() {
|
||||
NonBlocking input(0);
|
||||
std::string buffer;
|
||||
bool eof = false, quit = false;
|
||||
auto until = Clock::time_point::max();
|
||||
while (!interrupted) {
|
||||
Tick();
|
||||
output_.Pump();
|
||||
PumpVideo();
|
||||
if (!quit) {
|
||||
char bytes[8192];
|
||||
const auto n = Read(0, bytes, sizeof(bytes));
|
||||
if (n > 0)
|
||||
buffer.append(bytes, size_t(n));
|
||||
else if (!n)
|
||||
eof = true;
|
||||
else if (errno != EAGAIN && errno != EINTR)
|
||||
eof = true;
|
||||
if (buffer.size() > 65536) {
|
||||
Error(nullptr, "MESSAGE_TOO_LARGE");
|
||||
quit = true;
|
||||
}
|
||||
for (int count = 0; count < 16 && !quit; ++count) {
|
||||
const auto end = buffer.find('\n');
|
||||
if (end == std::string::npos)
|
||||
break;
|
||||
auto request = ParseBoundedJson(buffer.substr(0, end));
|
||||
buffer.erase(0, end + 1);
|
||||
quit = Handle(request);
|
||||
}
|
||||
if (eof && buffer.find('\n') == std::string::npos) {
|
||||
if (!buffer.empty())
|
||||
Error(nullptr, "INCOMPLETE_MESSAGE");
|
||||
quit = true;
|
||||
}
|
||||
}
|
||||
if (quit && until == Clock::time_point::max()) {
|
||||
Stop("CANCELLED");
|
||||
session_.Disconnect();
|
||||
PumpVideo();
|
||||
until = Clock::now() + std::chrono::seconds(2);
|
||||
}
|
||||
if (quit && (output_.Empty() || Clock::now() >= until))
|
||||
break;
|
||||
WaitIo(quit ? -1 : 0, output_.Empty() ? -1 : output_fd(), 5);
|
||||
}
|
||||
return interrupted ? 130 : 0;
|
||||
}
|
||||
void SetOutputFd(int fd) { output_fd_ = fd; }
|
||||
|
||||
private:
|
||||
int output_fd() const { return output_fd_; }
|
||||
json Status() const {
|
||||
auto result = session_.Status();
|
||||
result["video"] = {{"decode_path", decode_path_}, {"decoder", decoder_}};
|
||||
result["video_output"]["enabled"] = video_ && video_->Enabled();
|
||||
result["video_output"]["protocol"] =
|
||||
decode_path_ == "minirtc" ? "nv12/1" : "h264-annexb/1";
|
||||
if (video_) result["video_output"].update(video_->Status());
|
||||
if (decode_path_ == "minirtc") {
|
||||
result["video_output"]["received"] = frame_ ? frame_->frame_id : 0;
|
||||
result["video_output"]["waiting_keyframe"] = false;
|
||||
result["video_output"].erase("has_sps");
|
||||
result["video_output"].erase("has_pps");
|
||||
}
|
||||
result["transport_state"] = result["state"];
|
||||
result["geometry_epoch"] = epoch_;
|
||||
result["human_paused"] = paused_;
|
||||
result["control_allowed"] = allowed_;
|
||||
result["frame_age_ms"] = frame_ ? json(Age(frame_->queued)) : json(nullptr);
|
||||
result["source_size"] =
|
||||
frame_ ? json{{"width", frame_->width}, {"height", frame_->height}}
|
||||
: json(nullptr);
|
||||
if (session_.Connected())
|
||||
result["state"] = frame_ && frame_->generation == session_.Generation()
|
||||
? "ready"
|
||||
: "waiting_frame";
|
||||
return result;
|
||||
}
|
||||
void PumpVideo() {
|
||||
if (!video_) return;
|
||||
const auto generation = session_.Generation();
|
||||
const bool connected = session_.Connected();
|
||||
const bool decoded = decode_path_ == "minirtc";
|
||||
if (generation != video_generation_ || connected != video_connected_) {
|
||||
video_->Bind(generation, decoded, connected);
|
||||
video_generation_ = generation;
|
||||
video_connected_ = connected;
|
||||
last_video_frame_ = 0;
|
||||
}
|
||||
const bool enabled = video_->Enabled();
|
||||
const bool key = video_->TakeKeyframeRequest();
|
||||
if (enabled != video_enabled_ || key) {
|
||||
session_.SetVideoOutput(enabled && !decoded, key && !decoded);
|
||||
video_enabled_ = enabled;
|
||||
last_video_frame_ = 0;
|
||||
}
|
||||
if (!enabled) return;
|
||||
if (decoded) {
|
||||
if (frame_ && frame_->generation == generation && Age(frame_->queued) <= 2000 &&
|
||||
frame_->frame_id != last_video_frame_) {
|
||||
video_->Submit(frame_);
|
||||
last_video_frame_ = frame_->frame_id;
|
||||
}
|
||||
} else {
|
||||
for (int count = 0; count < 8; ++count) {
|
||||
auto frame = session_.TakeEncodedVideo();
|
||||
if (!frame) break;
|
||||
video_->Submit(std::move(*frame));
|
||||
}
|
||||
}
|
||||
}
|
||||
void Reply(const json &id, json value) {
|
||||
output_.Put({{"id", id}, {"ok", true}, {"result", std::move(value)}});
|
||||
}
|
||||
void Error(const json &id, const std::string &code,
|
||||
bool side_effect = false) {
|
||||
output_.Put({{"id", id},
|
||||
{"ok", false},
|
||||
{"error",
|
||||
{{"code", code},
|
||||
{"retryable", !side_effect && (code == "FRAME_TIMEOUT" ||
|
||||
code == "BUSY")},
|
||||
{"side_effect_possible", side_effect}}}});
|
||||
}
|
||||
void Release() {
|
||||
const bool enabled = session_.Status()["control"];
|
||||
session_.SetControl(false);
|
||||
if (enabled && allowed_ && !paused_ && session_.Connected())
|
||||
session_.SetControl(true);
|
||||
}
|
||||
void FinishAction(const char *error = nullptr) {
|
||||
if (!action_)
|
||||
return;
|
||||
Release();
|
||||
json result = {
|
||||
{"status", error ? (action_->submitted ? "outcome_unknown"
|
||||
: "failed_before_submit")
|
||||
: "submitted"},
|
||||
{"operation_id", action_->operation},
|
||||
{"generation", action_->generation},
|
||||
{"after_frame_id", frame_ ? frame_->frame_id : 0},
|
||||
{"side_effect_possible", action_->submitted},
|
||||
{"retryable", false}};
|
||||
if (error)
|
||||
result["error"] = error;
|
||||
for (auto &record : records_)
|
||||
if (record.operation == action_->operation) {
|
||||
record.result = result;
|
||||
break;
|
||||
}
|
||||
Reply(action_->id, result);
|
||||
action_.reset();
|
||||
}
|
||||
void Stop(const char *code) {
|
||||
FinishAction(code);
|
||||
for (const auto &capture : captures_)
|
||||
Error(capture.id, code);
|
||||
captures_.clear();
|
||||
if (encoding_ && !encoding_->cancelled) {
|
||||
Error(encoding_->capture.id, code);
|
||||
encoding_->cancelled = true;
|
||||
}
|
||||
observations_.clear();
|
||||
preview_ = json();
|
||||
}
|
||||
void Tick() {
|
||||
session_.Tick();
|
||||
if (auto f = session_.TakeFrame()) {
|
||||
if (!frame_ || f->generation != frame_->generation ||
|
||||
f->width != frame_->width || f->height != frame_->height) {
|
||||
++epoch_;
|
||||
observations_.clear();
|
||||
preview_ = json();
|
||||
}
|
||||
frame_ = std::make_shared<DecodedFrame>(std::move(*f));
|
||||
}
|
||||
if (frame_ && frame_->generation != session_.Generation()) {
|
||||
frame_.reset();
|
||||
++epoch_;
|
||||
observations_.clear();
|
||||
preview_ = json();
|
||||
}
|
||||
if (action_) {
|
||||
if (!session_.Connected() || !frame_ || paused_ || !allowed_ ||
|
||||
action_->generation != session_.Generation() ||
|
||||
action_->epoch != epoch_)
|
||||
FinishAction("SESSION_CHANGED");
|
||||
while (action_ && action_->next < action_->steps.size() &&
|
||||
Age(action_->started) >= action_->steps[action_->next].delay) {
|
||||
const auto &step = action_->steps[action_->next++];
|
||||
action_->submitted =
|
||||
true; // A failed send may still have reached the peer.
|
||||
const bool sent =
|
||||
step.kind == 0
|
||||
? session_.Pointer(step.x, step.y, step.flag, step.wheel)
|
||||
: step.kind == 1 ? session_.Key(step.key, step.down)
|
||||
: session_.Text(step.text);
|
||||
if (!sent) {
|
||||
FinishAction("ACTION_OUTCOME_UNKNOWN");
|
||||
break;
|
||||
}
|
||||
action_->submitted = true;
|
||||
}
|
||||
if (action_ && action_->next == action_->steps.size())
|
||||
FinishAction();
|
||||
}
|
||||
if (encoding_ &&
|
||||
encoding_->result.wait_for(Ms(0)) == std::future_status::ready) {
|
||||
auto job = std::move(encoding_);
|
||||
try {
|
||||
auto image = job->result.get();
|
||||
if (!job->cancelled) {
|
||||
if (!frame_ || job->epoch != epoch_ ||
|
||||
job->frame->generation != session_.Generation())
|
||||
Error(job->capture.id, "STALE_OBSERVATION");
|
||||
else if (Age(job->frame->queued) > 2000 ||
|
||||
Clock::now() > job->capture.deadline)
|
||||
Error(job->capture.id, "FRAME_TIMEOUT");
|
||||
else {
|
||||
const auto &f = *job->frame;
|
||||
json meta = {
|
||||
{"generation", f.generation},
|
||||
{"geometry_epoch", job->epoch},
|
||||
{"frame_id", f.frame_id},
|
||||
{"display_index", session_.Status()["display"]},
|
||||
{"source_size", {{"width", f.width}, {"height", f.height}}},
|
||||
{"source_rect",
|
||||
{{"x", image.region.x},
|
||||
{"y", image.region.y},
|
||||
{"width", image.region.width},
|
||||
{"height", image.region.height}}},
|
||||
{"image_size",
|
||||
{{"width", image.width}, {"height", image.height}}},
|
||||
{"frame_age_ms", Age(f.queued)}};
|
||||
json result = {{"mime_type", "image/png"},
|
||||
{"data", ImageBase64(image.png)},
|
||||
{"metadata", meta}};
|
||||
if (job->capture.preview) {
|
||||
preview_ = result;
|
||||
preview_time_ = f.queued;
|
||||
preview_edge_ =
|
||||
Number(job->capture.request, "max_edge", 64, 960, 960);
|
||||
} else {
|
||||
const auto id = "obs_" + std::to_string(f.generation) + "_" +
|
||||
std::to_string(++observation_sequence_);
|
||||
result["metadata"]["observation_id"] = id;
|
||||
if (observations_.size() >= 32)
|
||||
observations_.pop_front();
|
||||
observations_.push_back(
|
||||
{id, f.generation, job->epoch, image.width, image.height,
|
||||
int(f.width), int(f.height), image.region, Clock::now()});
|
||||
}
|
||||
Reply(job->capture.id, std::move(result));
|
||||
}
|
||||
}
|
||||
} catch (...) {
|
||||
if (!job->cancelled)
|
||||
Error(job->capture.id, "IMAGE_ENCODING_FAILED");
|
||||
}
|
||||
}
|
||||
for (auto it = captures_.begin(); it != captures_.end();) {
|
||||
if (it->request.contains("generation") &&
|
||||
Sequence(it->request, "generation") != session_.Generation()) {
|
||||
Error(it->id, "STALE_OBSERVATION");
|
||||
it = captures_.erase(it);
|
||||
} else if (Clock::now() > it->deadline || session_.Finished()) {
|
||||
Error(it->id, session_.Finished() ? "NOT_READY" : "FRAME_TIMEOUT");
|
||||
it = captures_.erase(it);
|
||||
} else
|
||||
++it;
|
||||
}
|
||||
if (!encoding_ && !captures_.empty() && frame_ &&
|
||||
Age(frame_->queued) <= 2000) {
|
||||
auto it = std::find_if(
|
||||
captures_.begin(), captures_.end(), [&](const Capture &c) {
|
||||
return !c.request.contains("after_frame_id") ||
|
||||
frame_->frame_id > Sequence(c.request, "after_frame_id");
|
||||
});
|
||||
if (it != captures_.end()) {
|
||||
auto capture = *it;
|
||||
const auto capture_id = capture.id;
|
||||
captures_.erase(it);
|
||||
try {
|
||||
Encode(std::move(capture));
|
||||
} catch (...) {
|
||||
Error(capture_id, "INVALID_REGION");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
void Encode(Capture capture) {
|
||||
ImageRegion region;
|
||||
if (capture.request.contains("region")) {
|
||||
const auto &r = capture.request["region"];
|
||||
Fields(r, {"x", "y", "width", "height"});
|
||||
region = {Number(r, "x", 0, int(frame_->width) - 1, -1),
|
||||
Number(r, "y", 0, int(frame_->height) - 1, -1),
|
||||
Number(r, "width", 1, frame_->width, -1),
|
||||
Number(r, "height", 1, frame_->height, -1)};
|
||||
if (region.x < 0 || region.y < 0 || region.width < 1 ||
|
||||
region.height < 1 || region.x > int(frame_->width) - region.width ||
|
||||
region.y > int(frame_->height) - region.height)
|
||||
throw std::invalid_argument("INVALID_REGION");
|
||||
}
|
||||
const int edge =
|
||||
Number(capture.request, "max_edge", 64, capture.preview ? 960 : 2048,
|
||||
capture.preview ? 960 : 1280);
|
||||
auto job = std::make_unique<Encoding>();
|
||||
job->capture = capture;
|
||||
job->frame = frame_;
|
||||
job->epoch = epoch_;
|
||||
job->result = std::async(std::launch::async, [frame = frame_, region, edge,
|
||||
preview = capture.preview] {
|
||||
return EncodeFrameImage(*frame, region, edge,
|
||||
preview ? 256 * 1024 : 2 * 1024 * 1024);
|
||||
});
|
||||
encoding_ = std::move(job);
|
||||
}
|
||||
void CaptureFrame(const json &j, const json &id, bool preview) {
|
||||
Fields(j, {"id", "method", "region", "max_edge", "after_frame_id",
|
||||
"generation", "timeout_ms"});
|
||||
if (!session_.Connected())
|
||||
throw std::invalid_argument("NOT_READY");
|
||||
Number(j, "max_edge", 64, preview ? 960 : 2048, preview ? 960 : 1280);
|
||||
if (preview && j.contains("region"))
|
||||
throw std::invalid_argument("INVALID_REQUEST");
|
||||
if (j.contains("after_frame_id") && !j.contains("generation"))
|
||||
throw std::invalid_argument("GENERATION_REQUIRED");
|
||||
Sequence(j, "after_frame_id");
|
||||
if (j.contains("generation") &&
|
||||
Sequence(j, "generation") != session_.Generation())
|
||||
throw std::invalid_argument("STALE_OBSERVATION");
|
||||
if (preview && frame_ && j.contains("after_frame_id") &&
|
||||
Sequence(j, "after_frame_id") >= frame_->frame_id) {
|
||||
Reply(id, {{"unchanged", true},
|
||||
{"generation", frame_->generation},
|
||||
{"frame_id", frame_->frame_id},
|
||||
{"frame_age_ms", Age(frame_->queued)},
|
||||
{"stale", Age(frame_->queued) > 2000}});
|
||||
return;
|
||||
}
|
||||
if (preview && frame_ && !preview_.is_null() &&
|
||||
Age(preview_time_) <= 2000 &&
|
||||
preview_edge_ == Number(j, "max_edge", 64, 960, 960) &&
|
||||
preview_["metadata"]["generation"] == frame_->generation &&
|
||||
preview_["metadata"]["frame_id"] == frame_->frame_id) {
|
||||
auto cached = preview_;
|
||||
cached["metadata"]["frame_age_ms"] = Age(preview_time_);
|
||||
Reply(id, cached);
|
||||
return;
|
||||
}
|
||||
if (captures_.size() >= 16)
|
||||
throw std::invalid_argument("BUSY");
|
||||
Capture c{id, j, preview,
|
||||
Clock::now() + Ms(Number(j, "timeout_ms", 0, 5000, 2000))};
|
||||
if (preview)
|
||||
captures_.push_back(c);
|
||||
else
|
||||
captures_.push_front(c);
|
||||
}
|
||||
const Observation &Observe(const json &j) {
|
||||
auto name = String(j, "observation_id");
|
||||
auto it = std::find_if(observations_.begin(), observations_.end(),
|
||||
[&](const Observation &o) { return o.id == name; });
|
||||
if (it == observations_.end() || it->generation != session_.Generation() ||
|
||||
it->epoch != epoch_ || Age(it->created) > 30000)
|
||||
throw std::invalid_argument("STALE_OBSERVATION");
|
||||
return *it;
|
||||
}
|
||||
void StartAction(const json &j, const json &id) {
|
||||
Fields(j, {"id", "method", "operation_id", "observation_id", "action"});
|
||||
const auto operation = String(j, "operation_id");
|
||||
const auto arguments = json{
|
||||
{"observation_id", j.at("observation_id")},
|
||||
{"action",
|
||||
j.at("action")}}.dump();
|
||||
for (const auto &r : records_)
|
||||
if (r.operation == operation) {
|
||||
if (r.arguments != arguments)
|
||||
throw std::invalid_argument("OPERATION_ID_CONFLICT");
|
||||
Reply(id, r.result);
|
||||
return;
|
||||
}
|
||||
if (!allowed_ || paused_ || !session_.Status()["control"].get<bool>())
|
||||
throw std::invalid_argument("CONTROL_DENIED");
|
||||
if (!session_.Connected() || !frame_)
|
||||
throw std::invalid_argument("NOT_READY");
|
||||
if (action_)
|
||||
throw std::invalid_argument("BUSY");
|
||||
const auto &o = Observe(j);
|
||||
const auto &a = j.at("action");
|
||||
const auto type = String(a, "type", 32);
|
||||
auto running = std::make_unique<RunningAction>();
|
||||
running->id = id;
|
||||
running->operation = operation;
|
||||
running->generation = o.generation;
|
||||
running->epoch = o.epoch;
|
||||
auto point = [&](const json &p, int delay, int flag = 0, int wheel = 0) {
|
||||
auto coordinate = [&](const char *key, int size, int offset, int span,
|
||||
int source) {
|
||||
if (!p.contains(key) || !p[key].is_number())
|
||||
throw std::invalid_argument("INVALID_COORDINATE");
|
||||
const auto value = p[key].get<double>();
|
||||
if (!std::isfinite(value) || value < 0 || value >= size)
|
||||
throw std::invalid_argument("INVALID_COORDINATE");
|
||||
return float(std::clamp((offset + (value + .5) * span / size) / source,
|
||||
0.0, 1.0));
|
||||
};
|
||||
Step step;
|
||||
step.delay = delay;
|
||||
step.flag = flag;
|
||||
step.wheel = wheel;
|
||||
step.x =
|
||||
coordinate("x", o.width, o.region.x, o.region.width, o.source_width);
|
||||
step.y = coordinate("y", o.height, o.region.y, o.region.height,
|
||||
o.source_height);
|
||||
running->steps.push_back(std::move(step));
|
||||
};
|
||||
auto key = [&](int code, bool down, int delay) {
|
||||
Step step;
|
||||
step.kind = 1;
|
||||
step.key = code;
|
||||
step.down = down;
|
||||
step.delay = delay;
|
||||
running->steps.push_back(std::move(step));
|
||||
};
|
||||
if (type == "move" || type == "click" || type == "double_click") {
|
||||
Fields(a, type == "move"
|
||||
? std::initializer_list<const char *>{"type", "x", "y"}
|
||||
: std::initializer_list<const char *>{"type", "x", "y",
|
||||
"button"});
|
||||
point(a, 0);
|
||||
if (type != "move") {
|
||||
const auto button =
|
||||
a.contains("button") ? String(a, "button", 8) : "left";
|
||||
const int flag = button == "left" ? 1
|
||||
: button == "right" ? 3
|
||||
: button == "middle" ? 5
|
||||
: -1;
|
||||
if (flag < 0)
|
||||
throw std::invalid_argument("INVALID_BUTTON");
|
||||
point(a, 10, flag);
|
||||
point(a, 40, flag + 1);
|
||||
if (type == "double_click") {
|
||||
point(a, 110, flag);
|
||||
point(a, 140, flag + 1);
|
||||
}
|
||||
}
|
||||
} else if (type == "scroll") {
|
||||
Fields(a, {"type", "x", "y", "direction", "ticks"});
|
||||
const auto direction = String(a, "direction", 8);
|
||||
if (direction != "up" && direction != "down" && direction != "left" &&
|
||||
direction != "right")
|
||||
throw std::invalid_argument("INVALID_DIRECTION");
|
||||
const int ticks = Number(a, "ticks", 1, 10, 1);
|
||||
point(a, 0);
|
||||
point(a, 10, direction == "left" || direction == "right" ? 8 : 7,
|
||||
(direction == "up" || direction == "right" ? 120 : -120) * ticks);
|
||||
} else if (type == "drag") {
|
||||
Fields(a, {"type", "path", "duration_ms"});
|
||||
const auto &path = a.at("path");
|
||||
if (!path.is_array() || path.size() < 2 || path.size() > 128)
|
||||
throw std::invalid_argument("INVALID_PATH");
|
||||
const int duration = Number(a, "duration_ms", 100, 2000, 500);
|
||||
for (const auto &p : path)
|
||||
Fields(p, {"x", "y"});
|
||||
point(path.front(), 0);
|
||||
point(path.front(), 10, 1);
|
||||
for (size_t i = 1; i < path.size(); ++i)
|
||||
point(path[i], 10 + int(i * duration / (path.size() - 1)));
|
||||
point(path.back(), duration + 30, 2);
|
||||
} else if (type == "key_press") {
|
||||
Fields(a, {"type", "key"});
|
||||
const auto code = KeyCode(String(a, "key", 16));
|
||||
key(code, true, 0);
|
||||
key(code, false, 40);
|
||||
} else if (type == "hotkey") {
|
||||
Fields(a, {"type", "keys"});
|
||||
const auto &keys = a.at("keys");
|
||||
if (!keys.is_array() || keys.size() < 2 || keys.size() > 6)
|
||||
throw std::invalid_argument("INVALID_KEYS");
|
||||
std::set<int> seen;
|
||||
std::vector<int> codes;
|
||||
for (const auto &item : keys) {
|
||||
if (!item.is_string())
|
||||
throw std::invalid_argument("INVALID_KEY");
|
||||
int code = KeyCode(item.get<std::string>());
|
||||
if (!seen.insert(code).second)
|
||||
throw std::invalid_argument("DUPLICATE_KEY");
|
||||
codes.push_back(code);
|
||||
}
|
||||
int delay = 0;
|
||||
for (int code : codes) {
|
||||
key(code, true, delay);
|
||||
delay += 15;
|
||||
}
|
||||
delay += 30;
|
||||
for (auto it = codes.rbegin(); it != codes.rend(); ++it) {
|
||||
key(*it, false, delay);
|
||||
delay += 15;
|
||||
}
|
||||
} else if (type == "type_text") {
|
||||
Fields(a, {"type", "text"});
|
||||
if (!session_.Status()["text_input"].get<bool>())
|
||||
throw std::invalid_argument("TEXT_INPUT_UNSUPPORTED");
|
||||
Step step;
|
||||
step.kind = 2;
|
||||
step.text = String(a, "text", 16384);
|
||||
running->steps.push_back(std::move(step));
|
||||
} else
|
||||
throw std::invalid_argument("UNSUPPORTED_ACTION");
|
||||
while (records_.size() >= 128 ||
|
||||
(!records_.empty() && Age(records_.front().created) > 300000))
|
||||
records_.pop_front();
|
||||
records_.push_back(
|
||||
{operation,
|
||||
arguments,
|
||||
{{"status", "in_progress"}, {"operation_id", operation}},
|
||||
Clock::now()});
|
||||
running->started = Clock::now();
|
||||
action_ = std::move(running);
|
||||
}
|
||||
std::string Password() {
|
||||
if (password_fd_ < 3 || password_fd_ > 1024)
|
||||
throw std::invalid_argument("CREDENTIAL_CHANNEL_INVALID");
|
||||
std::string secret;
|
||||
const auto deadline = Clock::now() + std::chrono::seconds(5);
|
||||
while (!interrupted && Clock::now() < deadline) {
|
||||
if (!WaitReadable(password_fd_, 10))
|
||||
continue;
|
||||
char c;
|
||||
const auto n = Read(password_fd_, &c, 1);
|
||||
if (!n || (n == 1 && c == '\n'))
|
||||
return secret;
|
||||
if (n < 0) {
|
||||
if (errno == EINTR || errno == EAGAIN)
|
||||
continue;
|
||||
throw std::invalid_argument("CREDENTIAL_CHANNEL_FAILED");
|
||||
}
|
||||
if (!c || secret.size() >= 256)
|
||||
throw std::invalid_argument("INVALID_CREDENTIAL");
|
||||
secret += c;
|
||||
}
|
||||
std::fill(secret.begin(), secret.end(), '\0');
|
||||
throw std::invalid_argument("CREDENTIAL_TIMEOUT");
|
||||
}
|
||||
bool Handle(const json &j) {
|
||||
json id = nullptr;
|
||||
try {
|
||||
if (!j.is_object() || !j.contains("id") || !j["id"].is_number_integer() ||
|
||||
j["id"].get<int64_t>() < 0)
|
||||
throw std::invalid_argument("INVALID_REQUEST");
|
||||
id = j["id"];
|
||||
const auto method = String(j, "method", 32);
|
||||
if (method == "hello") {
|
||||
Fields(j, {"id", "method", "protocol"});
|
||||
if (String(j, "protocol") != "crossdesk-automation/1")
|
||||
throw std::invalid_argument("VERSION_MISMATCH");
|
||||
hello_ = true;
|
||||
Reply(id, {{"protocol", "crossdesk-automation/1"},
|
||||
{"image_formats", {"image/png"}},
|
||||
{"video_protocols", {"h264-annexb/1", "nv12/1"}},
|
||||
{"video_decoders", {"auto", "software"}},
|
||||
{"video_transport", "loopback-websocket/1"},
|
||||
{"action_completion", "submitted"},
|
||||
{"max_request_bytes", 65536},
|
||||
{"max_response_bytes", kMaxResponse},
|
||||
{"operation_cache_entries", 128},
|
||||
{"operation_cache_ms", 300000},
|
||||
{"human_pause", true}});
|
||||
return false;
|
||||
}
|
||||
if (!hello_)
|
||||
throw std::invalid_argument("HELLO_REQUIRED");
|
||||
if (method == "connect") {
|
||||
Fields(j, {"id", "method", "device", "display", "mode",
|
||||
"decode_path", "decoder"});
|
||||
if (connected_once_)
|
||||
throw std::invalid_argument("NEW_PROCESS_REQUIRED");
|
||||
const auto device = String(j, "device", 9);
|
||||
if (device.size() != 9 ||
|
||||
device.find_first_not_of("0123456789") != std::string::npos)
|
||||
throw std::invalid_argument("INVALID_DEVICE");
|
||||
const auto mode = j.contains("mode") ? String(j, "mode", 8) : "observe";
|
||||
if (mode != "observe" && mode != "control")
|
||||
throw std::invalid_argument("INVALID_MODE");
|
||||
options_.display = Number(j, "display", 0, 63, 0);
|
||||
const auto path = j.contains("decode_path")
|
||||
? String(j, "decode_path", 16) : "browser";
|
||||
const auto decoder = j.contains("decoder")
|
||||
? String(j, "decoder", 16) : "auto";
|
||||
if ((path != "browser" && path != "minirtc") ||
|
||||
(decoder != "auto" && decoder != "software"))
|
||||
throw std::invalid_argument("INVALID_VIDEO_CONFIG");
|
||||
decode_path_ = path;
|
||||
decoder_ = decoder;
|
||||
options_.hardware_acceleration = decoder != "software";
|
||||
auto password = Password();
|
||||
connected_once_ = true;
|
||||
paused_ = false;
|
||||
allowed_ = mode == "control";
|
||||
session_.Connect(options_, device, std::move(password));
|
||||
session_.SetControl(allowed_);
|
||||
} else if (method == "screenshot" || method == "preview") {
|
||||
CaptureFrame(j, id, method == "preview");
|
||||
return false;
|
||||
} else if (method == "action") {
|
||||
StartAction(j, id);
|
||||
return false;
|
||||
} else if (method == "control") {
|
||||
Fields(j, {"id", "method", "enabled"});
|
||||
const bool enable = Flag(j, "enabled");
|
||||
if (enable && (!allowed_ || paused_))
|
||||
throw std::invalid_argument("CONTROL_DENIED");
|
||||
if (!enable)
|
||||
FinishAction("CONTROL_DISABLED");
|
||||
session_.SetControl(enable);
|
||||
} else if (method == "viewer_control") {
|
||||
Fields(j, {"id", "method", "operation"});
|
||||
const auto operation = String(j, "operation", 16);
|
||||
if (operation == "pause") {
|
||||
paused_ = true;
|
||||
FinishAction("USER_PAUSED");
|
||||
session_.SetControl(false);
|
||||
} else if (operation == "resume") {
|
||||
if (!allowed_)
|
||||
throw std::invalid_argument("CONTROL_DENIED");
|
||||
paused_ = false;
|
||||
session_.SetControl(true);
|
||||
} else if (operation == "disconnect") {
|
||||
Stop("CANCELLED");
|
||||
session_.Disconnect();
|
||||
} else
|
||||
throw std::invalid_argument("INVALID_REQUEST");
|
||||
} else if (method == "display") {
|
||||
Fields(j, {"id", "method", "index"});
|
||||
Stop("SESSION_CHANGED");
|
||||
if (!session_.SelectDisplay(Number(j, "index", 0, 63, -1)))
|
||||
throw std::invalid_argument("DISPLAY_UNAVAILABLE");
|
||||
frame_.reset();
|
||||
++epoch_;
|
||||
} else if (method == "cancel") {
|
||||
Fields(j, {"id", "method", "request_id"});
|
||||
if (!j.contains("request_id") || !j["request_id"].is_number_integer())
|
||||
throw std::invalid_argument("INVALID_REQUEST");
|
||||
const auto target = j["request_id"];
|
||||
if (action_ && action_->id == target)
|
||||
FinishAction("CANCELLED");
|
||||
if (encoding_ && !encoding_->cancelled &&
|
||||
encoding_->capture.id == target) {
|
||||
encoding_->cancelled = true;
|
||||
Error(target, "CANCELLED");
|
||||
}
|
||||
for (auto it = captures_.begin(); it != captures_.end();) {
|
||||
if (it->id == target) {
|
||||
Error(target, "CANCELLED");
|
||||
it = captures_.erase(it);
|
||||
} else
|
||||
++it;
|
||||
}
|
||||
} else if (method == "disconnect" || method == "exit") {
|
||||
Fields(j, {"id", "method"});
|
||||
Stop("CANCELLED");
|
||||
session_.Disconnect();
|
||||
frame_.reset();
|
||||
} else if (method == "video_open") {
|
||||
Fields(j, {"id", "method", "ticket", "view", "view_generation",
|
||||
"session", "origin", "apps"});
|
||||
if (!video_) throw std::invalid_argument("VIDEO_CHANNEL_UNAVAILABLE");
|
||||
if (!session_.Connected()) throw std::invalid_argument("NOT_READY");
|
||||
PumpVideo();
|
||||
video_->Grant({String(j, "ticket", 32), String(j, "view", 32),
|
||||
String(j, "session", 32), String(j, "origin", 256),
|
||||
Sequence(j, "view_generation"), Flag(j, "apps")});
|
||||
Reply(id, {{"url", video_->Url()}, {"transport", "loopback-websocket/1"},
|
||||
{"protocol", decode_path_ == "minirtc" ? "nv12/1" : "h264-annexb/1"},
|
||||
{"decoder", decoder_}, {"expires_in_ms", 15000}});
|
||||
return false;
|
||||
} else if (method == "video_revoke") {
|
||||
Fields(j, {"id", "method", "view"});
|
||||
if (video_) video_->Revoke(j.contains("view") ? String(j, "view", 32) : "");
|
||||
} else if (method == "status")
|
||||
Fields(j, {"id", "method"});
|
||||
else
|
||||
throw std::invalid_argument("UNSUPPORTED_METHOD");
|
||||
PumpVideo();
|
||||
Reply(id, Status());
|
||||
return method == "exit";
|
||||
} catch (const std::invalid_argument &e) {
|
||||
Error(id, e.what());
|
||||
} catch (const json::exception &) {
|
||||
Error(id, "INVALID_REQUEST");
|
||||
} catch (...) {
|
||||
Error(id, "OPERATION_FAILED");
|
||||
}
|
||||
return false;
|
||||
}
|
||||
ControllerSession session_;
|
||||
Responses output_;
|
||||
SessionOptions options_;
|
||||
std::string decode_path_ = "browser", decoder_ = "auto";
|
||||
int password_fd_, output_fd_ = -1;
|
||||
bool hello_ = false, connected_once_ = false, allowed_ = false,
|
||||
paused_ = false;
|
||||
uint64_t epoch_ = 0, observation_sequence_ = 0;
|
||||
std::shared_ptr<const DecodedFrame> frame_;
|
||||
std::deque<Observation> observations_;
|
||||
std::deque<Capture> captures_;
|
||||
std::unique_ptr<Encoding> encoding_;
|
||||
std::unique_ptr<RunningAction> action_;
|
||||
std::deque<Record> records_;
|
||||
json preview_;
|
||||
int preview_edge_ = 960;
|
||||
Clock::time_point preview_time_;
|
||||
std::unique_ptr<VideoServer> video_;
|
||||
uint64_t video_generation_ = 0, last_video_frame_ = 0;
|
||||
bool video_connected_ = false, video_enabled_ = false;
|
||||
};
|
||||
} // namespace
|
||||
int Agent(int output_fd, SessionOptions options, int password_fd,
|
||||
bool video_websocket) {
|
||||
Automation automation(output_fd, std::move(options), password_fd, video_websocket);
|
||||
automation.SetOutputFd(output_fd);
|
||||
return automation.Run();
|
||||
}
|
||||
} // namespace crossdesk::cli
|
||||
@@ -0,0 +1,17 @@
|
||||
/*
|
||||
* @Author: DI JUNKUN
|
||||
* @Date: 2026-11-02
|
||||
* Copyright (c) 2026 by DI JUNKUN, All Rights Reserved.
|
||||
*/
|
||||
|
||||
#ifndef _AGENT_H_
|
||||
#define _AGENT_H_
|
||||
|
||||
#include <controller_session.h>
|
||||
|
||||
namespace crossdesk::cli {
|
||||
int Agent(int output_fd, SessionOptions options, int password_fd,
|
||||
bool video_websocket = false);
|
||||
}
|
||||
|
||||
#endif
|
||||
@@ -0,0 +1,110 @@
|
||||
# Native automation protocol
|
||||
|
||||
`crossdesk-cli agent --stdio --password-fd 3 --video-ws` is the private native adapter used by CrossDesk MCP. It is separate from the existing terminal-oriented `bridge` protocol. This is **not MCP on stdin**: the TypeScript service owns the public MCP transport and tool schemas.
|
||||
|
||||
Optional native arguments are `--server HOST`, `--port PORT`, `--quality PRESET` and `--log-dir DIRECTORY`. The caller creates the log directory. Diagnostic stdout is redirected to stderr; only protocol responses use the preserved output descriptor. Passwords travel over descriptor 3, terminated by newline or EOF, and never appear in JSON requests. `--video-ws` starts an authenticated, native loopback WebSocket listener on an ephemeral port. There is no video file descriptor. Agent mode selects the high-frame-rate preview tier at up to 1920 × 1080 by default; negotiated remote capabilities and network conditions determine the delivered rate. Each process accepts one connection attempt; retry authentication with a new process and credential pipe.
|
||||
|
||||
## Envelope
|
||||
|
||||
One UTF-8 JSON object per line, maximum request 64 KiB and nesting depth 32. Use nonnegative integer request IDs. Responses may complete out of order; correlate by ID.
|
||||
|
||||
```json
|
||||
{"id":1,"method":"hello","protocol":"crossdesk-automation/1"}
|
||||
{"id":1,"ok":true,"result":{"protocol":"crossdesk-automation/1","image_formats":["image/png"],"action_completion":"submitted","human_pause":true,"max_request_bytes":65536,"max_response_bytes":4194304,"operation_cache_entries":128,"operation_cache_ms":300000}}
|
||||
```
|
||||
|
||||
Unknown fields and malformed arguments are rejected. Error responses contain a stable code, retryability and whether input may already have been submitted:
|
||||
|
||||
```json
|
||||
{"id":2,"ok":false,"error":{"code":"CONTROL_DENIED","retryable":false,"side_effect_possible":false}}
|
||||
```
|
||||
|
||||
The output queue is bounded to 64 messages and 8 MiB; a blocked output pipe terminates after 10 seconds without progress. EOF, signals and `exit` disconnect and release held input. A caller that needs responses must keep stdin open until they arrive.
|
||||
|
||||
## Methods
|
||||
|
||||
| Method | Parameters in addition to `id`/`method` |
|
||||
| --- | --- |
|
||||
| `hello` | `protocol: "crossdesk-automation/1"` |
|
||||
| `connect` | `device` (9 digits), `mode: "observe" | "control"`, `display` (0–63), optional `decode_path: "browser" / "minirtc"`, `decoder: "auto" / "software"` |
|
||||
| `status` | none |
|
||||
| `screenshot` | optional `region: {x,y,width,height}`, `max_edge` (64–2048), `after_frame_id`, `generation`, `timeout_ms` (0–5000) |
|
||||
| `video_open` | `ticket`, `view`, `view_generation`, `session`, `origin`, `apps`; requires `--video-ws` and a connected session |
|
||||
| `video_revoke` | optional `view`; omit to revoke all video grants and sockets |
|
||||
| `preview` | optional `max_edge` (64–960), `after_frame_id`, `generation`, `timeout_ms` (0–5000); no region |
|
||||
| `action` | `operation_id`, `observation_id`, `action` |
|
||||
| `control` | `enabled` boolean; cannot override observe-only permission or human pause |
|
||||
| `viewer_control` | `operation: "pause" | "resume" | "disconnect"`; restricted to trusted UI by the MCP adapter |
|
||||
| `display` | `index` (0–63) |
|
||||
| `cancel` | `request_id`; cancels queued/in-flight captures or input sequences |
|
||||
| `disconnect`, `exit` | none; `exit` also ends the process |
|
||||
|
||||
`connect` returns immediately after initiating transport. Poll `status` until `state == "ready"`; this requires a frame from the current display generation. Status includes `transport_state`, `geometry_epoch`, `frame_age_ms`, `source_size`, `control_allowed`, `human_paused`, host platform, displays and transport diagnostics.
|
||||
|
||||
## Live video
|
||||
|
||||
`doctor --json` advertises `automation_video_transport: "loopback-websocket/1"`, `automation_video_protocols: ["h264-annexb/1", "nv12/1"]`, and `automation_video_decoders: ["auto", "software"]`. `hello` repeats these without the `automation_` prefix. The MCP adapter requires the transport capability, so an old pipe-only CLI is rejected. Node spawns only stdin, stdout, stderr, and the credential pipe; live video never enters Node.
|
||||
|
||||
`connect.decode_path` defaults to `browser` (compressed H.264), or selects `minirtc` (decoded NV12). `connect.decoder` defaults to `auto` (hardware preferred, software when unavailable); `software` selects the native software decoder. Both paths maintain native screenshot frames with the selected decoder. These choices are fixed for the connection. Status exposes `video` and the reported `native_video_decoder` backend name. PNG encoding runs only when requested. The legacy `preview` image method remains available to private protocol callers but is not used by MCP panels.
|
||||
|
||||
### Authorization and transport
|
||||
|
||||
The trusted adapter generates 24 random bytes as a 32-character base64url `ticket` and sends `video_open` over private JSON-RPC. `view` and `session` are also 32-character base64url IDs; `view_generation` is positive. `origin` is the Node panel's exact `http://127.0.0.1:PORT` origin. `apps` permits an opaque (`null`) or HTTPS MCP Apps origin, but still requires the same one-time ticket. No password is included.
|
||||
|
||||
The response contains `{url, transport, protocol, decoder, expires_in_ms:15000}`. Node supplies the ticket to the authorized UI. The browser opens the returned `ws://127.0.0.1:PORT/api/video` directly and sends `{"ticket":"..."}` as its first text message within 3 seconds. Tickets expire after 15 seconds, are consumed once, and are bound to a view/session plus the native display generation. The native service checks the exact Host and path, RFC 6455 version, and authorized Origin. It does not serve HTTP content or accept remote input commands. Logs are disabled for this listener; tickets never go in URLs or model-visible results.
|
||||
|
||||
The listener uses its own I/O thread, at most 16 WebSocket candidates and 8 authorized viewers. Client messages are limited to 1024 bytes. Revocation, disconnect, native generation changes, or shutdown cancel sockets and invalidate tickets, including partially sent frames. Pause only disables Agent control; viewing continues. The adapter explicitly revokes a removed or replaced view, and terminates the native session if revocation cannot be confirmed. Native processes own ephemeral ports, so panel CSP allows `ws://127.0.0.1:*`, restricted to the literal loopback address; the control HTTP origin remains exact.
|
||||
|
||||
### Binary messages and flow control
|
||||
|
||||
Each binary WebSocket message contains one complete frame, **without the old pipe length prefix**. Integers are big-endian. For `h264-annexb/1`:
|
||||
|
||||
| Offset | Type | Value |
|
||||
| --- | --- | --- |
|
||||
| 0 | u32 | `0x43445631` (`CDV1`) |
|
||||
| 4 | u32 | Display/connection generation |
|
||||
| 8 | u32 | Receive sequence; gaps require a new keyframe |
|
||||
| 12 | u64 | Local monotonic receive timestamp in microseconds |
|
||||
| 20 | u32 | Flags: 1 = IDR keyframe, 0 = delta |
|
||||
| 24 | bytes | Annex B H.264; keyframes include SPS and PPS |
|
||||
|
||||
Access units are capped at 4 MiB. Both the receive tap and the I/O handoff are bounded to 60 frames / 8 MiB. Overflow discards dependent frames until an IDR. After receiving each message the browser sends `{"type":"ack","generation":N,"sequence":N}` directly to the native socket, in order. This acknowledges receipt, not decode, presentation, or remote execution. Per-viewer unacknowledged output is capped at 8 frames; at 1 MiB outstanding, further frames are skipped until space is available (a single keyframe may exceed that threshold). Dropping a dependent picture requires a new keyframe; `{"type":"keyframe"}` requests recovery, rate-limited to once per second.
|
||||
|
||||
For `nv12/1`, magic is `0x43444e31` (`CDN1`). Offsets 4, 8, 12 and 20 remain generation, decoded sequence, local monotonic decoded-frame queue time in microseconds, and flags (always 1). Offsets 24 and 28 are u32 width and height; offset 32 starts tightly packed Y followed by interleaved UV. Dimensions must be even, at most 8192 per axis / 16777216 pixels. Message size must equal `32 + width * height * 3 / 2` (at most 25165856 bytes). Color is limited-range BT.601, matching the existing PNG conversion. Sequence gaps are allowed.
|
||||
|
||||
NV12 retains one latest picture in the I/O handoff and at most one unacknowledged picture per viewer. New pictures are skipped for a slow viewer without blocking others. Each prepared WebSocket message is immutable and shared between recipients, avoiding per-viewer payload copies. There is still a native copy into the WebSocket payload and raw pixel transfer to the browser; this is not GPU zero-copy. A viewer whose oldest outstanding frame remains unacknowledged for 10 seconds is disconnected. Hidden/unloaded pages close their native socket. The browser replaces pending raw pictures and does not decode H.264 in this mode.
|
||||
|
||||
`status.video_output` includes `transport`, `viewers`, I/O handoff `queued` / `queued_bytes`, aggregate `unacknowledged_bytes`, and `dropped` (handoff replacements plus per-viewer congestion skips). These diagnostics contain no video payload. The original receive-tap SPS/PPS and keyframe state remain available for H.264.
|
||||
|
||||
The MiniRTC `Params` structure appends `on_receive_encoded_video`, changing its ABI size. Rebuild consumers and MiniRTC together; zero-initialize `Params`. The callback borrows frame bytes on the receive/decode queue and must not retain the pointer. Returning nonzero requests a remote keyframe. No remote wire-protocol change is required.
|
||||
|
||||
## Images and observations
|
||||
|
||||
Images return `{mime_type:"image/png",data:"<base64>",metadata:{...}}`. Metadata includes `generation`, `geometry_epoch`, `frame_id`, `display_index`, `source_size`, `source_rect`, `image_size` and `frame_age_ms`; only screenshots create `observation_id`. Preview images cannot authorize model input.
|
||||
|
||||
NV12 frames must be tightly packed, even dimensions, at most 8192 per axis and 16 million pixels. Conversion/PNG encoding runs on one worker with no `ControllerSession` calls. The owner loop continues processing cancellation and input release. Screenshots have priority over queued previews. An encoding from an obsolete generation or geometry epoch is discarded. Frames older than two seconds are not used for a new image. PNG limits are 2 MiB for observations and 256 KiB for previews; the encoder reduces dimensions when necessary.
|
||||
|
||||
`after_frame_id` requires `generation`. A screenshot waits up to `timeout_ms` for a newer frame; a preview can return `{unchanged:true, frame_age_ms, stale, generation, frame_id}`. Neither path treats an image as an input acknowledgement.
|
||||
|
||||
## Actions
|
||||
|
||||
All coordinates refer to the immutable returned observation image. The native layer maps them through its source rectangle to normalized remote coordinates, without terminal layout padding. Coordinates must satisfy `0 <= x < image_width` and `0 <= y < image_height`. Observations expire after 30 seconds, retain at most 32 entries, and are invalidated by geometry/display generation changes.
|
||||
|
||||
```json
|
||||
{"id":10,"method":"action","operation_id":"click-1","observation_id":"obs_2_1","action":{"type":"click","x":120,"y":80,"button":"left"}}
|
||||
```
|
||||
|
||||
Supported payloads:
|
||||
|
||||
- `move`: `x`, `y`.
|
||||
- `click`, `double_click`: `x`, `y`, optional `button` (`left`, `right`, `middle`).
|
||||
- `scroll`: `x`, `y`, `direction` (`up`, `down`, `left`, `right`), optional `ticks` (1–10).
|
||||
- `drag`: `path` of 2–128 `{x,y}` points, optional `duration_ms` (100–2000).
|
||||
- `key_press`: `key`; `hotkey`: 2–6 unique `keys`.
|
||||
- `type_text`: `text`, at most 16 KiB UTF-8; requires negotiated remote text-input capability.
|
||||
|
||||
Only one input sequence runs at once. Timed steps are advanced by the owner loop; cancellation, disconnect, display changes and pause release held buttons/keys. A failed send is conservatively considered potentially submitted.
|
||||
|
||||
Results use `status: submitted | in_progress | failed_before_submit | outcome_unknown`, with `operation_id`, `retryable`, `side_effect_possible` and, for completed actions, `generation`/`after_frame_id`. The same operation ID and arguments retrieve the cached result; different arguments conflict. Retention is bounded to 128 entries, with entries older than five minutes eligible for removal on a new action. Never retry an uncertain effect with a new ID. `submitted` does not claim remote execution.
|
||||
|
||||
The native process trusts its parent. The parent must authenticate panel-only `viewer_control` calls and protect descriptor 3; this private pipe does not isolate arbitrary programs running as the same OS user.
|
||||
@@ -12,7 +12,7 @@
|
||||
| connect | device:9位数字字符串, display:0..63（默认0） | 从专用 password fd 读取至换行/EOF（最多256字节、等待5秒），开始异步注册/连接；ok不是远端已连接 |
|
||||
| status | 无 | 返回当前状态 |
|
||||
| control | enabled:boolean | 显式启停控制；关闭时释放输入 |
|
||||
| input | data:string，最多16KiB | 终端原始按键/UTF-8/bracketed paste/SGR鼠标；只读时 c 开启控制、q/Ctrl+C 退出，d/鼠标点击进入细节，+/-/滚轮缩放，方向键或 i 上、j 左、k 下、l 右平移，f 全景；Ctrl+] 打开本地命令，随后 c 选择返回后的控制状态、q 退出或视图键连续调整画面，Esc/Ctrl+C 手动返回；控制模式中 Ctrl+T/Ctrl+C 传给远端，本地模式中再按 Ctrl+] 手动返回，并在返回控制模式时传递远端 Ctrl+] |
|
||||
| input | data:string，最多16KiB | 终端原始按键/UTF-8/bracketed paste/SGR鼠标；只读时 c 开启控制、q/Ctrl+C 退出，鼠标点击进入细节，+/-/滚轮以鼠标位置缩放，方向键或 i 上、j 左、k 下、l 右平移，f 恢复全景并重置倍率与平移；Ctrl+] 打开本地命令，随后 c 选择返回后的控制状态、q 退出或视图键连续调整画面，Esc/Ctrl+C 手动返回；控制模式中 Ctrl+T/Ctrl+C 传给远端，本地模式中再按 Ctrl+] 手动返回，并在返回控制模式时传递远端 Ctrl+] |
|
||||
| display | index:0..63 | 选择已广告的零起始显示器，提升 generation |
|
||||
| ack | frame_id、generation：非负整数 | 确认宿主消费帧；匹配当前等待帧才开放下一帧 |
|
||||
| disconnect | 无 | 释放输入、结束当前媒体会话 |
|
||||
|
||||
@@ -7,6 +7,8 @@ CLI 从独立开发目录整合到 CrossDesk，入口、打包脚本、npm 启
|
||||
- `ControllerSession` 的流名称、HostInformation 和输入参考 Android `rtc_bridge.cpp`；身份使用单 Peer 的 `cli` 临时注册协议；终端适配不依赖 JNI、Surface 或 GuiRuntime。
|
||||
- 帧缓存遵守 `MiniRtcVideoFrame` 回调借用契约，直接复制到有界 CPU 存储，不跨回调保留 native 指针。
|
||||
- PreviewSubscription 复用 wire 实现与对端 capability 字段，不新增被控端协议。
|
||||
- MCP 实时面板新增 MiniRTC 的 `on_receive_encoded_video` 回调，转发解码前的完整 H.264 Annex B 帧；回调及关键帧请求由当前子模块源码提供。`Params` 新增字段改变 ABI 大小，CLI 与 MiniRTC 必须一起重建；发布时同步记录包含该改动的子模块提交。
|
||||
- 自动化图像编码复用 MiniRTC 已有的 libyuv 2025.8.14，并通过 Xmake 固定静态 libpng v1.6.59；libpng 的压缩依赖沿用平台 zlib。新增依赖的许可证和对应源码也应纳入正式发行的依赖材料。
|
||||
|
||||
MiniRTC 已在提交 `b9ee1fa7fe038fdf771f5dec1b1d561d78273f15` 中合入凭据日志脱敏：Join、排队 Join、取消 Join、Leave 四个日志点只记录 `@` 前的 ID，不记录密码。`apps/cli/scripts/prepare-deps.sh` 仅按主仓库记录初始化子模块，不再应用独立补丁；更新 MiniRTC 时只需更新子模块提交。共享子模块中的脱敏同样适用于桌面程序；该修改没有改变鉴权、TLS、TURN 或媒体协议。
|
||||
|
||||
|
||||
@@ -57,11 +57,11 @@ Windows 的可见传统控制台优先读取当前字体尺寸；Windows Termina
|
||||
|
||||
缩放在完整输出边界清理旧画面。上限 240 列、100 画面行；默认顶部预留一行状态栏、底部预留两行快捷键，另预留一列避免自动换行。顶部状态栏用反色区分，左侧为控制模式和连接进度，右侧为画面参数及网络信息；底部两行分别为会话和视图快捷键。视频和等待提示从顶部状态栏下方开始，鼠标坐标按视频区域映射，点击状态栏不会传给远端。按键分组使用统一列宽和左右留白，窄窗口自动减少列数、合并按键组并省略次要参数，累计帧数和输出字节数保留在日志中。控制模式下仅提示本地命令入口及后续操作，进入本地命令模式后才显示平移和缩放按键；终端高度不足时优先收起底部提示行。
|
||||
|
||||
快捷键行使用 `KEYS |`、`VIEW |` 等固定宽度标题，标题和按键组之间用竖线分隔；键名用 `<c>`、`<+/->` 等标签标识。宽窗口中的分组分隔线按列对齐，窄窗口保留标题和完整按键标签，按可用宽度合并或减少分组。
|
||||
快捷键行使用 `KEYS |`、`VIEW |` 等固定宽度标题，标题和按键组之间用竖线分隔；键名用 `<c>`、`<+/->` 等标签标识。`Zoom at mouse` 表示鼠标锚点缩放，`Full view` / `Full` 表示恢复全景，`Pan` 表示平移。只读模式提示滚轮缩放，宽窗口还显示点击进入细节；本地命令模式提示返回、控制切换及键盘视图操作，宽窗口显示鼠标选择锚点和 `=` 放大。宽窗口中的分组分隔线按列对齐，窄窗口保留完整按键标签，优先保留缩放和全景，按可用宽度合并或减少分组。
|
||||
|
||||
顶部状态栏以整数显示实时帧率（例如 `FPS 30`），约每秒按最近统计区间内完整输出到终端的帧数计算；没有新画面输出时更新为 `FPS 0`。该数值是实际提交速率，不是 `--fps` 目标值或终端 GPU 呈现频率；窄窗口优先省略画面参数和网络信息，保留帧率。
|
||||
|
||||
ANSI 在完整桌面缩略图中仍无法保留普通字号的文字。先增加终端列数（缩小终端字体或扩大窗口），阅读局部时用 `--zoom native`。只读全景中点击目标位置可直接进入该区域的细节模式；按 `d` 则以当前中心进入。细节模式依据视口和字符宽高比选择源区域，避免为了完整装入桌面而把文字缩小；ANSI 视野较小，借助方向键平移阅读。`--zoom 1..32` 是相对整桌面的倍率；在细节模式中 `+/-` 是相对原始细节尺寸的倍率。画面缩放完全在本地进行，控制鼠标坐标映射回原始桌面的正确区域。
|
||||
ANSI 在完整桌面缩略图中仍无法保留普通字号的文字。先增加终端列数（缩小终端字体或扩大窗口），阅读局部时用 `--zoom native`。只读时可用 `+/-` 或滚轮以鼠标位置缩放，也可点击目标位置直接进入该区域的细节模式。细节模式依据视口和字符宽高比选择源区域，避免为了完整装入桌面而把文字缩小；ANSI 视野较小，借助方向键平移阅读。`--zoom 1..32` 是相对整桌面的倍率；在细节模式中 `+/-` 是相对原始细节尺寸的倍率。画面缩放完全在本地进行，控制鼠标坐标映射回原始桌面的正确区域。
|
||||
|
||||
auto 模式发送 Kitty 查询与主设备属性查询（DA1），等待最多 250ms；优先选择 Kitty，否则在 DA1 的能力参数中找到 `4` 时选择 Sixel，再回退为 ANSI。显式 `kitty` 或 `sixel` 模式探测失败时报错，不强制向未知终端输出图像。tmux/screen 中 auto 使用 ANSI，显式像素模式报不支持；不修改用户终端配置。
|
||||
|
||||
@@ -90,6 +90,8 @@ crossdesk-cli 123456789 --renderer ansi --fps 8
|
||||
|
||||
`--fps 1..30` 是本地输出目标。交互 CLI 的 ANSI、Sixel、内联 Kitty 和共享内存 Kitty 默认均为 30 FPS；共享内存失败回退时保持该目标。bridge 仍按宿主协议默认使用 8 FPS。
|
||||
|
||||
独立 CLI 与 bridge 的远端预览统一请求清晰度优先（保持分辨率），包括默认 30 FPS、显式 `--fps` 和自动切换预览帧率。提高帧率不会将编码策略切回帧率优先；实际分辨率仍受画质上限、源画面尺寸、被控端支持及编码／传输条件限制。
|
||||
|
||||
未显式指定 `--fps` 时，远端预览从 30 FPS 开始，根据完整一帧从转换开始到输出完成的耗时自动选择 8／15／30 FPS 请求；共享内存的上传确认等待也计入，等待源帧的空闲时间不计入。使用平滑估计、不同升降档阈值和至少 8 个新帧样本，持续变慢 2 秒才降档、持续恢复 4 秒才升档，保留供帧余量。15 FPS 仅在对端声明支持时使用；预览未获接受时不追加自动调节。视口变化、显示器切换和共享内存回退会重置耗时估计。显式 `--fps` 固定本地目标和对应的远端预览请求，关闭自动调节；例如 `--fps 30` 可用于与自动策略对比。
|
||||
|
||||
本地调度只在有新帧或需要重绘时消耗刷新时隙；短暂迟到保持原刷新节奏，长时间阻塞后跳过旧时隙，不连续补画过时帧。目标帧率不代表终端实际显示帧率。
|
||||
@@ -125,12 +127,13 @@ macOS 自带 Terminal 还需要在当前窗口勾选“显示 → 允许鼠标
|
||||
| 退出 CLI | `q` 或 `Ctrl+C` | `Ctrl+]`，再按 `q` |
|
||||
| 进入本地命令模式 | `?` 或 `Ctrl+]` | `Ctrl+]` |
|
||||
| 放大／缩小 | `+` / `=` / `-` | `Ctrl+]`，再按对应键 |
|
||||
| 原始细节／完整画面 | `d`（detail）/ `f`（fit），相邻字母键 | `Ctrl+]`，再按对应键 |
|
||||
| 恢复完整画面 | `f`，重置倍率、平移和细节模式 | `Ctrl+]`，再按 `f` |
|
||||
| 鼠标查看 | 滚轮以鼠标位置缩放；点击进入原始细节 | 进入本地命令模式后仅更新缩放锚点，使用缩放键调整 |
|
||||
| 平移 | 方向键或 `i` 上、`j` 左、`k` 下、`l` 右 | `Ctrl+]`，再按对应键 |
|
||||
|
||||
进入本地命令模式时暂停远端输入，并释放已记录按键／鼠标。执行缩放、平移、`c` 或未知命令后均保持此模式；`c` 切换手动返回后是否启用远端控制，状态栏显示 `CONTROL PAUSED` 或 `READ ONLY`。`Esc` 或 `Ctrl+C` 手动返回所选模式；`q` 退出 CLI。再次按 `Ctrl+]` 也会返回，并在所选模式为控制时向远端发送一次 `Ctrl+]`。命令模式中的鼠标事件被忽略，bracketed paste 内容被丢弃，均不会退出模式或触发快捷键。窗口失焦或输入缓冲区溢出时继续保持本地命令模式，但将返回模式设为只读。控制模式下普通 `c/q/?`、`Ctrl+T`、`Ctrl+C` 和单独 `Esc` 均传给远端。目前不支持重绑定。
|
||||
进入本地命令模式时暂停远端输入，并释放已记录按键／鼠标。执行缩放、平移、`c` 或未知命令后均保持此模式；`c` 切换手动返回后是否启用远端控制，状态栏显示 `CONTROL PAUSED` 或 `READ ONLY`。`Esc` 或 `Ctrl+C` 手动返回所选模式；`q` 退出 CLI。再次按 `Ctrl+]` 也会返回，并在所选模式为控制时向远端发送一次 `Ctrl+]`。命令模式中的鼠标事件仅更新本地缩放锚点，不执行点击、拖动或滚轮操作；bracketed paste 内容被丢弃，均不会退出模式或触发快捷键。窗口失焦或输入缓冲区溢出时继续保持本地命令模式，但将返回模式设为只读。控制模式下普通 `c/q/?`、`Ctrl+T`、`Ctrl+C` 和单独 `Esc` 均传给远端。目前不支持重绑定。
|
||||
|
||||
只读时，`d` 或在画面中单击进入该位置的原始细节模式；`+` / `=` 放大、`-` 缩小（1～32倍），滚轮以鼠标指向为中心调整倍率。方向键或 `i/j/k/l` 分别向上/左/下/右平移，`f` 恢复完整画面。黑边点击被忽略。这些按键和鼠标在控制模式下仍传给远端；进入本地命令模式后可连续调整视图，直至手动返回。放大、平移和只读鼠标定位不会向远端发送操作，也适用于静止画面。
|
||||
只读时，在画面中单击进入该位置的原始细节模式；`+` / `=` 放大、`-` 缩小（1～32倍）。缩放键与滚轮均以鼠标位置为锚点，尽量保持鼠标下的画面内容在原位置；到达桌面边缘时限制裁剪范围。没有收到鼠标位置，或鼠标位于黑边、状态栏时，缩放键沿用当前中心；窗口失焦会清除鼠标位置。方向键或 `i/j/k/l` 分别向上/左/下/右平移，`f` 恢复完整画面并重置倍率、平移和细节模式。黑边点击被忽略。这些按键和鼠标在控制模式下仍传给远端；进入本地命令模式后可移动鼠标选择锚点并连续按缩放键调整视图，直至手动返回。放大、平移和只读鼠标定位不会向远端发送操作，也适用于静止画面。
|
||||
|
||||
支持 SGR 和传统 `CSI M` 鼠标报告，包括移动、左右中键、拖动和滚轮；完整缓冲跨读取分片的传统报告，坐标字节不会作为按键执行。启动时依次启用普通、拖动和全移动报告，兼容只支持部分跟踪模式的终端；退出时关闭这些模式。传统报告的坐标上限为 223，较大窗口应使用支持 SGR 的终端。协议格式依据 [XTerm 鼠标协议](https://invisible-island.net/xterm/ctlseqs/ctlseqs.html#h2-Mouse-Tracking)。
|
||||
|
||||
@@ -153,7 +156,7 @@ export CROSSDESK_CLI="$PWD/build/macosx/arm64/release/crossdesk-cli"
|
||||
pi -e ./apps/cli/plugins/pi/index.ts
|
||||
```
|
||||
|
||||
在 Pi 输入 `/crossdesk` 或 `/crossdesk 123456789`。先输入 ID，再在自定义遮蔽组件输入密码。扩展使用独立 fd 3 写密码，随后关闭；不写配置、聊天、工具结果或模型上下文。查看组件与独立 CLI 使用相同的 `Ctrl+]` 命令前缀和上述模式按键，由 bridge 统一解析。插件默认请求 high 预览并显示 ANSI 象限字符行，支持同样的只读细节/放大/平移键，颜色深度依据终端环境回退，由 Pi 自身布局和刷新；不直接占用 TTY，不发送 Kitty 图形或自行打开鼠标报告。
|
||||
在 Pi 输入 `/crossdesk` 或 `/crossdesk 123456789`。先输入 ID，再在自定义遮蔽组件输入密码。扩展使用独立 fd 3 写密码，随后关闭；不写配置、聊天、工具结果或模型上下文。查看组件与独立 CLI 使用相同的 `Ctrl+]` 命令前缀和上述模式按键，由 bridge 统一解析。插件默认请求 high 预览并显示 ANSI 象限字符行，支持同样的缩放、全景和平移键，颜色深度依据终端环境回退，由 Pi 自身布局和刷新；不直接占用 TTY，不发送 Kitty 图形或自行打开鼠标报告。
|
||||
|
||||
Pi 扩展读取宿主的字符尺寸，并通过宿主终端写入／输入监听接口查询字符尺寸或完整文本区尺寸，每两秒及窗口变化时刷新；尺寸回复不会传给远端。得到的宽高比通过 bridge 的 `viewport.cell_aspect` 传入，比例变化时重绘静止画面，并丢弃仍在途的旧比例帧。自动检测不可用或不准确时，可在启动 Pi 前设置环境变量 `CROSSDESK_CELL_ASPECT=0.45`（宽／高，允许 0.2～2；`auto` 恢复自动检测）。视图关闭或 bridge 停止时移除查询定时器和输入监听。
|
||||
|
||||
@@ -180,6 +183,6 @@ Pi 扩展读取宿主的字符尺寸，并通过宿主终端写入／输入监
|
||||
- `doctor --json` 是环境报告，不会联网登录、主动探测 Kitty / Sixel 或证明远端可连接。 `input_shortcuts=local_commands_v2` 表示支持本文的持续本地命令模式；启动日志同时记录实际执行路径，以便区分不同工程的构建产物。
|
||||
- 画面比例异常时查看 `client.log` 摘要的 `terminal_geometry`：`cell_aspect` 为实际采用的字符宽高比，`reported` 表示收到有效的终端尺寸报告，`manual_aspect` 表示使用了手动覆盖。`doctor` 中的比例和像素尺寸只反映系统 TTY 信息，不主动查询终端，可能与交互连接自动检测后的值不同。
|
||||
|
||||
第一轮无音频播放或音频订阅 API，SSH 服务器上的声音不会传回本地。无后台守护进程、共享媒体 Broker、IP 直连、MCP 或自动截图。其他宿主可以使用 [私有桥接](bridge-protocol.md)，没有可靠自定义 UI 时打开独立终端或给出播放器命令。
|
||||
终端模式无音频播放或音频订阅 API，SSH 服务器上的声音不会传回本地。无后台守护进程、共享媒体 Broker 或 IP 直连。终端宿主可以使用 [私有桥接](bridge-protocol.md)，没有可靠自定义 UI 时打开独立终端。Agent 的 PNG 截图和结构化操作由独立的 [自动化模式](automation-protocol.md) 和 [MCP 服务](../../mcp/README.md) 提供。
|
||||
|
||||
每个播放器/桥接进程只拥有自己的一个媒体会话，不向模型消息发送视频。
|
||||
|
||||
@@ -22,12 +22,19 @@
|
||||
|
||||
新增回归代码、基准、假传输和验证产物均位于工程外临时目录，未向工程或子模块添加测试。此次完整构建与运行验证覆盖 macOS arm64，未重新进行 Linux/Windows 实机验证。
|
||||
|
||||
## 控制按键调整（2026-10-30）
|
||||
## 鼠标锚点缩放与快捷键提示（2026-11-02）
|
||||
|
||||
- 移除 `d` 本地快捷键，保留鼠标点击进入细节及 `--zoom native`。`f` 恢复完整桌面并重置倍率、平移和细节模式；`+/-`、`=` 和只读滚轮使用鼠标锚点缩放。
|
||||
- 工程外 ASan/UBSan 检查通过 900 次缩放几何检查，覆盖 ASCII、ANSI half/quadrant、Kitty/Sixel 视口、全景/细节、连续输入、边缘限制、SGR/传统鼠标报告、失焦与模式切换；确认 `d` 不再改变本地视图，控制模式仍转发普通字母。
|
||||
- 四种控制/本地模式组合、1～240 列共 960 组状态栏布局检查通过，覆盖完整按键标签、缩放/全景提示及模式可用操作。Pi 提示通过 Node.js 语法检查；没有进行真实 Pi TUI 或远端设备连接验证。
|
||||
- 新增验证代码及产物均保存在工程外临时目录。
|
||||
|
||||
## 控制按键调整（2026-10-30，历史记录）
|
||||
|
||||
- 当前桌面与 CLI 同时启用的 debug 配置下，`xmake build crossdesk-cli`、`xmake run crossdesk-cli doctor --json` 和命令帮助检查通过。
|
||||
- 工程外 ASan/UBSan 检查覆盖本地命令前缀、只读快捷键、远端 Ctrl+T/Ctrl+C/Ctrl+]、鼠标释放、取消命令、粘贴隔离、UTF-8/CSI 分片、失焦与退出；原有会话／渲染检查通过。
|
||||
- 平移字母键采用 `i` 上、`j` 左、`k` 下、`l` 右；工程外 ASan/UBSan 检查确认四个方向与普通／应用模式方向键一致，控制模式中普通字母仍传给远端。
|
||||
- 细节／全景采用相邻的 `d`／`f`；工程外 ASan/UBSan 检查确认细节保留中心、全景复位缩放与中心、前缀命令和鼠标点击行为正确，控制模式中普通字母仍传给远端。
|
||||
- 当时细节／全景采用相邻的 `d`／`f`（`d` 已于 2026-11-02 移除）；工程外 ASan/UBSan 检查确认细节保留中心、全景复位缩放与中心、前缀命令和鼠标点击行为正确，控制模式中普通字母仍传给远端。
|
||||
- 实际二进制 bridge 验证命令状态、错误恢复和退出响应；生产 view 配合假传输的 PTY 检查验证快捷键、模式提示、隐藏密码和终端恢复。实际二进制的 bridge 与 PTY 回归同样通过。
|
||||
- Pi 1.4.0 SDK 严格类型检查和模拟组件运行通过，覆盖控制按键转发、命令提示、取消、缩放、退出与 bridge 启动失败后的关闭操作。
|
||||
|
||||
|
||||
+30
-10
@@ -1,4 +1,5 @@
|
||||
#include "terminal_io.h"
|
||||
#include "agent.h"
|
||||
#include <algorithm>
|
||||
#include <cerrno>
|
||||
#include <clocale>
|
||||
@@ -608,7 +609,7 @@ int View(SessionOptions options, const std::string &device,
|
||||
: sixel ? "SIXEL"
|
||||
: detail == AnsiDetail::Quadrant ? "QUAD"
|
||||
: "HALF") +
|
||||
(input.native ? " | Detail x" : " | Fit x") +
|
||||
(input.native ? " | Detail x" : " | Zoom x") +
|
||||
std::to_string(int(input.zoom)) +
|
||||
(last_frame ? " | " + std::to_string(last_frame->width) + "x" +
|
||||
std::to_string(last_frame->height)
|
||||
@@ -1034,14 +1035,19 @@ const char *help = R"(crossdesk-cli <9-digit-device-id> [--renderer auto|ansi|ki
|
||||
crossdesk-cli view <9-digit-device-id> [options] (compatibility alias)
|
||||
crossdesk-cli doctor [--json]
|
||||
crossdesk-cli bridge --stdio [--password-fd 3] [--server host] [--port 9099]
|
||||
crossdesk-cli agent --stdio [--password-fd 3] [--video-ws] [--server host] [--port 9099]
|
||||
|
||||
Connect by device ID: read-only and silent by default; hidden password prompt on the local terminal.
|
||||
Read-only: c enables control; q or Ctrl+C quits; ? shows local commands.
|
||||
Ctrl+] enters persistent local mode: c selects control on return; q quits; Esc/Ctrl+C returns.
|
||||
View keys: click/d native detail, wheel/+/- zoom, arrows/ijkl pan (i up, j left, k down, l right), f fit.
|
||||
View keys (read-only/local): +/- zoom at mouse (= also zooms in); f restores the full desktop and resets pan.
|
||||
Pan: arrows or i up, j left, k down, l right. Read-only mouse: wheel zooms at pointer; click opens native detail.
|
||||
Local mode: move the mouse to choose the zoom anchor; clicks and wheel do not change the view.
|
||||
Zoom uses the current center when no mouse position is available or the pointer is outside the picture.
|
||||
Local mode stays active after commands, paste or focus loss; remote input remains paused.
|
||||
Ctrl+T and Ctrl+C pass through in control mode; another Ctrl+] leaves local mode and forwards Ctrl+] if control is enabled.
|
||||
Default ANSI detail: quadrant. Default quality: high (1920x1080); ultra requests up to 3840x2160.
|
||||
Remote previews prioritize resolution at every frame rate.
|
||||
Cell aspect is character width/height: auto detects terminal geometry; use --cell-aspect to calibrate.
|
||||
Display index is zero-based. ANSI cannot reproduce full desktop text at fit scale.
|
||||
Auto rendering prefers Kitty, then Sixel, then ANSI. Explicit pixel renderers require a successful probe.
|
||||
@@ -1049,6 +1055,7 @@ Auto rendering prefers Kitty, then Sixel, then ANSI. Explicit pixel renderers re
|
||||
Explicit --fps keeps a fixed preview request. Kitty transport fallback keeps the local target unchanged.
|
||||
Preview requests above 15 use the negotiated interactive 30 FPS stream; lower requests use 1/8/15 FPS.
|
||||
bridge: private NDJSON v1, default 8 FPS, credentials on a separate inherited descriptor.
|
||||
agent: crossdesk-automation/1, structured actions and PNG screenshots; no terminal required.
|
||||
No IP direct connection, audio playback, clipboard or file transfer support.
|
||||
)";
|
||||
} // namespace
|
||||
@@ -1083,12 +1090,17 @@ int main(int argc, char **argv) {
|
||||
{"kitty", "probe_on_view"},
|
||||
{"sixel", "probe_on_view"},
|
||||
{"audio_playback", false},
|
||||
{"bridge_version", 1}};
|
||||
{"bridge_version", 1},
|
||||
{"automation_protocol", "crossdesk-automation/1"},
|
||||
{"automation_video", "h264-annexb/1"},
|
||||
{"automation_video_transport", "loopback-websocket/1"},
|
||||
{"automation_video_protocols", {"h264-annexb/1", "nv12/1"}},
|
||||
{"automation_video_decoders", {"auto", "software"}}};
|
||||
std::cout << d.dump(argc == 3 ? -1 : 2) << "\n";
|
||||
return 0;
|
||||
}
|
||||
int i = 2;
|
||||
if (command != "view" && command != "bridge") {
|
||||
if (command != "view" && command != "bridge" && command != "agent") {
|
||||
if (command.find_first_not_of("0123456789") != std::string::npos)
|
||||
throw std::invalid_argument(
|
||||
"expected a 9-digit device ID or a command; use --help");
|
||||
@@ -1101,6 +1113,8 @@ int main(int argc, char **argv) {
|
||||
double zoom = 1, cell_aspect = 0;
|
||||
bool control = false, stdio = false, native = false, fps_explicit = false;
|
||||
int secret_fd = 3;
|
||||
bool video_websocket = false;
|
||||
if (command == "agent") options.fps = 30;
|
||||
if (command == "view") {
|
||||
if (i >= argc)
|
||||
throw std::invalid_argument("device ID required");
|
||||
@@ -1120,8 +1134,10 @@ int main(int argc, char **argv) {
|
||||
std::string a = argv[i];
|
||||
if (a == "--control" && command == "view")
|
||||
control = true;
|
||||
else if (a == "--stdio" && command == "bridge")
|
||||
else if (a == "--stdio" && command != "view")
|
||||
stdio = true;
|
||||
else if (a == "--video-ws" && command == "agent")
|
||||
video_websocket = true;
|
||||
else {
|
||||
if (i + 1 >= argc)
|
||||
throw std::invalid_argument("option value missing");
|
||||
@@ -1130,6 +1146,8 @@ int main(int argc, char **argv) {
|
||||
options.host = value;
|
||||
else if (a == "--port")
|
||||
options.port = number(value, 1, 65535);
|
||||
else if (a == "--log-dir" && command == "agent")
|
||||
options.log_directory = value;
|
||||
else if (a == "--fps" && command == "view") {
|
||||
options.fps = number(value, 1, 30);
|
||||
fps_explicit = true;
|
||||
@@ -1154,7 +1172,7 @@ int main(int argc, char **argv) {
|
||||
} else if (a == "--zoom" && command == "view") {
|
||||
native = value == "native";
|
||||
zoom = native ? 1 : number(value, 1, 32);
|
||||
} else if (a == "--password-fd" && command == "bridge")
|
||||
} else if (a == "--password-fd" && command != "view")
|
||||
secret_fd = number(value, 3, 1024);
|
||||
else
|
||||
throw std::invalid_argument("unsupported option");
|
||||
@@ -1163,14 +1181,14 @@ int main(int argc, char **argv) {
|
||||
if (renderer != "auto" && renderer != "ansi" && renderer != "kitty" &&
|
||||
renderer != "sixel")
|
||||
throw std::invalid_argument("unknown renderer");
|
||||
if (command == "bridge" && !stdio)
|
||||
throw std::invalid_argument("bridge requires --stdio");
|
||||
if (command != "view" && !stdio)
|
||||
throw std::invalid_argument("automation and bridge require --stdio");
|
||||
Descriptor output(Duplicate(1));
|
||||
if (output.value < 0)
|
||||
throw std::runtime_error("cannot preserve output channel");
|
||||
// All library stdout writes go to diagnostics. Only Output uses the saved
|
||||
// fd.
|
||||
options.log_directory = Logs();
|
||||
if (options.log_directory.empty()) options.log_directory = Logs();
|
||||
if (command == "view") {
|
||||
Descriptor logs(OpenLog(options.log_directory + "/client.log"));
|
||||
if (logs.value < 0 || Redirect(logs.value, 2) < 0)
|
||||
@@ -1189,7 +1207,9 @@ int main(int argc, char **argv) {
|
||||
return command == "view" ? View(options, device, renderer, control,
|
||||
output.value, detail, zoom, native,
|
||||
cell_aspect, fps_explicit)
|
||||
: Bridge(output.value, options, secret_fd);
|
||||
: command == "agent"
|
||||
? Agent(output.value, options, secret_fd, video_websocket)
|
||||
: Bridge(output.value, options, secret_fd);
|
||||
} catch (const std::exception &e) {
|
||||
if (command == "view") {
|
||||
std::string msg = "crossdesk-cli: " + SafeText(e.what()) + "\r\n";
|
||||
|
||||
+13
-11
@@ -36,17 +36,19 @@ function main() {
|
||||
return;
|
||||
}
|
||||
const stdio = ['inherit', 'inherit', 'inherit'];
|
||||
if (args[0] === 'bridge') {
|
||||
const index = args.lastIndexOf('--password-fd');
|
||||
const fd = index === -1 ? 3 : Number(args[index + 1]);
|
||||
if (Number.isInteger(fd) && fd >= 3 && fd <= 1024) {
|
||||
try {
|
||||
const stat = fs.fstatSync(fd);
|
||||
if (stat.isFIFO() || stat.isSocket() || stat.isFile()) {
|
||||
while (stdio.length <= fd) stdio.push('ignore');
|
||||
stdio[fd] = fd;
|
||||
}
|
||||
} catch { /* The native bridge reports a missing credential descriptor. */ }
|
||||
if (args[0] === 'bridge' || args[0] === 'agent') {
|
||||
for (const [option, fallback] of [['--password-fd', 3]]) {
|
||||
const index = args.lastIndexOf(option);
|
||||
const fd = index === -1 ? fallback : Number(args[index + 1]);
|
||||
if (Number.isInteger(fd) && fd >= 3 && fd <= 1024) {
|
||||
try {
|
||||
const stat = fs.fstatSync(fd);
|
||||
if (stat.isFIFO() || stat.isSocket() || stat.isFile()) {
|
||||
while (stdio.length <= fd) stdio.push('ignore');
|
||||
stdio[fd] = fd;
|
||||
}
|
||||
} catch { /* Native code reports unavailable descriptors. */ }
|
||||
}
|
||||
}
|
||||
}
|
||||
const child = spawn(binary, args, { stdio, shell: false });
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@chunkdash/crossdesk-cli",
|
||||
"version": "0.1.5",
|
||||
"version": "0.2.0",
|
||||
"private": true,
|
||||
"description": "CrossDesk remote desktop controller for the terminal",
|
||||
"license": "GPL-3.0-only",
|
||||
|
||||
@@ -227,10 +227,10 @@ export default function crossdesk(pi: ExtensionAPI) {
|
||||
const ack = frame; frame = undefined;
|
||||
if (ack) queueMicrotask(() => send("ack", ack)); // Host consumed the frame; not a physical display timestamp.
|
||||
const hint = bridgeStopped ? "q / Esc / Ctrl+C close"
|
||||
: commandMode ? "Esc/Ctrl+C return | c toggle q quit | arrows/ijkl pan +/- zoom f fit d detail"
|
||||
: commandMode ? "Esc/Ctrl+C return | c toggle | q quit | +/- zoom | f full | arrows/ijkl pan"
|
||||
: control ? "Keys -> remote | Ctrl+] then c release, q quit"
|
||||
: "c control q quit ? help | i up j left k down l right | +/- zoom f fit d detail";
|
||||
return [truncateToWidth(`CrossDesk ${device} | ${status} ${native ? "detail" : "fit"} x${zoom} | Ctrl+] commands`, width),
|
||||
: "c control | q quit | +/- zoom | f full | arrows/ijkl pan | ? commands";
|
||||
return [truncateToWidth(`CrossDesk ${device} | ${status} ${native ? "detail" : "zoom"} x${zoom} | Ctrl+] commands`, width),
|
||||
...lines.slice(0, h).map(line => truncateToWidth(line, width)),
|
||||
truncateToWidth(`${hint}${error ? " | " + error : lines.length ? "" : " | Waiting for remote video…"}`, width)];
|
||||
},
|
||||
|
||||
@@ -84,6 +84,8 @@ function inspect(binary, target) {
|
||||
}
|
||||
const doctor = JSON.parse(run(binary, ['doctor', '--json'], { timeout: 15000 }));
|
||||
if (doctor.bridge_version !== 1) throw new Error('Native binary does not report the expected bridge version.');
|
||||
if (doctor.automation_protocol !== 'crossdesk-automation/1')
|
||||
throw new Error('Native binary does not report the expected automation protocol.');
|
||||
return info;
|
||||
}
|
||||
function pack(directory, output) {
|
||||
|
||||
@@ -0,0 +1,481 @@
|
||||
#include "video_server.h"
|
||||
|
||||
#include <atomic>
|
||||
#include <deque>
|
||||
#include <future>
|
||||
#include <map>
|
||||
#include <mutex>
|
||||
#include <thread>
|
||||
#include <websocketpp/config/asio_no_tls.hpp>
|
||||
#include <websocketpp/server.hpp>
|
||||
|
||||
namespace crossdesk::cli {
|
||||
namespace {
|
||||
using Clock = std::chrono::steady_clock;
|
||||
using namespace std::chrono_literals;
|
||||
using json = nlohmann::json;
|
||||
using Handle = websocketpp::connection_hdl;
|
||||
using Server = websocketpp::server<websocketpp::config::asio>;
|
||||
constexpr size_t kMaxEncoded = 4 * 1024 * 1024;
|
||||
constexpr size_t kMaxQueued = 8 * 1024 * 1024;
|
||||
|
||||
bool Token(const std::string &s) {
|
||||
return s.size() == 32 &&
|
||||
s.find_first_not_of("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvw"
|
||||
"xyz0123456789-_") == std::string::npos;
|
||||
}
|
||||
bool AppOrigin(const std::string &s) {
|
||||
return s == "null" ||
|
||||
(s.size() > 8 && s.size() <= 256 && s.compare(0, 8, "https://") == 0 &&
|
||||
s.find_first_of("/?#@\\ \t\r\n", 8) == std::string::npos);
|
||||
}
|
||||
bool BrowserOrigin(const std::string &s) {
|
||||
const std::string prefix = "http://127.0.0.1:";
|
||||
if (s.compare(0, prefix.size(), prefix) || s.size() > prefix.size() + 5 ||
|
||||
s.size() == prefix.size() ||
|
||||
s.find_first_not_of("0123456789", prefix.size()) != std::string::npos)
|
||||
return false;
|
||||
const auto port = std::stoul(s.substr(prefix.size()));
|
||||
return port > 0 && port <= 65535;
|
||||
}
|
||||
void Number(std::string &out, uint64_t value, int bytes) {
|
||||
for (int n = bytes - 1; n >= 0; --n)
|
||||
out.push_back(char(value >> (n * 8)));
|
||||
}
|
||||
} // namespace
|
||||
|
||||
struct VideoServer::State {
|
||||
struct Ticket {
|
||||
VideoGrant grant;
|
||||
Clock::time_point expires;
|
||||
};
|
||||
struct Receipt {
|
||||
uint32_t generation, sequence;
|
||||
size_t bytes;
|
||||
Clock::time_point sent;
|
||||
};
|
||||
struct Client {
|
||||
std::string origin, view, session;
|
||||
uint64_t view_generation = 0;
|
||||
bool authorized = false, waiting_key = true;
|
||||
size_t bytes = 0;
|
||||
Clock::time_point opened = Clock::now(), pinged = Clock::now();
|
||||
std::deque<Receipt> receipts;
|
||||
};
|
||||
struct Packet {
|
||||
std::shared_ptr<const DecodedFrame> raw;
|
||||
EncodedVideoFrame encoded;
|
||||
size_t Size() const {
|
||||
return raw ? 32 + raw->nv12.size() : 24 + encoded.data.size();
|
||||
}
|
||||
uint64_t Generation() const {
|
||||
return raw ? raw->generation : encoded.generation;
|
||||
}
|
||||
uint64_t Sequence() const { return raw ? raw->frame_id : encoded.sequence; }
|
||||
bool Key() const { return raw || encoded.key; }
|
||||
};
|
||||
|
||||
Server server;
|
||||
Server::timer_ptr timer;
|
||||
std::thread thread;
|
||||
std::string authority;
|
||||
std::map<Handle, Client, std::owner_less<Handle>> clients;
|
||||
std::map<std::string, Ticket> tickets;
|
||||
std::atomic<size_t> viewers{0}, outstanding{0}, dropped{0};
|
||||
std::atomic<bool> key_requested{false};
|
||||
uint64_t generation = 0;
|
||||
bool connected = false, decoded = false;
|
||||
Clock::time_point last_key_request{};
|
||||
|
||||
// One posted drain, not one posted handler per picture. Raw frames replace
|
||||
// each other; the compressed handoff is bounded and recovers at an IDR.
|
||||
mutable std::mutex mutex;
|
||||
std::deque<Packet> queue;
|
||||
size_t queued_bytes = 0;
|
||||
bool scheduled = false, wait_key = true;
|
||||
|
||||
State() {
|
||||
server.clear_access_channels(websocketpp::log::alevel::all);
|
||||
server.clear_error_channels(websocketpp::log::elevel::all);
|
||||
server.init_asio();
|
||||
server.set_open_handshake_timeout(3000);
|
||||
server.set_close_handshake_timeout(1000);
|
||||
server.set_pong_timeout(10000);
|
||||
server.set_max_message_size(1024);
|
||||
server.set_max_http_body_size(0);
|
||||
server.set_socket_init_handler([](Handle, asio::ip::tcp::socket &socket) {
|
||||
asio::error_code ec;
|
||||
socket.set_option(asio::ip::tcp::no_delay(true), ec);
|
||||
socket.set_option(asio::socket_base::send_buffer_size(256 * 1024), ec);
|
||||
});
|
||||
server.set_validate_handler([this](Handle h) {
|
||||
auto c = server.get_con_from_hdl(h);
|
||||
if (clients.size() >= 16 || c->get_resource() != "/api/video" ||
|
||||
c->get_request_header("Host") != authority ||
|
||||
c->get_request_header("Sec-WebSocket-Version") != "13" ||
|
||||
c->get_origin().empty() || c->get_origin().size() > 256) {
|
||||
c->set_status(websocketpp::http::status_code::forbidden);
|
||||
return false;
|
||||
}
|
||||
Client client;
|
||||
client.origin = c->get_origin();
|
||||
clients.emplace(h, std::move(client));
|
||||
return true;
|
||||
});
|
||||
server.set_message_handler([this](Handle h, Server::message_ptr message) {
|
||||
try {
|
||||
Message(h, message);
|
||||
} catch (...) {
|
||||
Drop(h);
|
||||
}
|
||||
});
|
||||
server.set_close_handler([this](Handle h) { Remove(h); });
|
||||
server.set_fail_handler([this](Handle h) { Remove(h); });
|
||||
server.set_pong_timeout_handler(
|
||||
[this](Handle h, const std::string &) { Drop(h); });
|
||||
server.listen(asio::ip::tcp::endpoint(asio::ip::address_v4::loopback(), 0));
|
||||
asio::error_code ec;
|
||||
authority =
|
||||
"127.0.0.1:" + std::to_string(server.get_local_endpoint(ec).port());
|
||||
if (ec)
|
||||
throw std::runtime_error("VIDEO_CHANNEL_UNAVAILABLE");
|
||||
server.start_accept();
|
||||
Maintain();
|
||||
thread = std::thread([this] { server.run(); });
|
||||
}
|
||||
~State() {
|
||||
Call([this] {
|
||||
websocketpp::lib::error_code ec;
|
||||
server.stop_listening(ec);
|
||||
if (timer)
|
||||
timer->cancel();
|
||||
Revoke("");
|
||||
});
|
||||
server.stop();
|
||||
thread.join();
|
||||
}
|
||||
template <class F> void Call(F f) {
|
||||
auto task = std::make_shared<std::packaged_task<void()>>(std::move(f));
|
||||
auto result = task->get_future();
|
||||
server.get_io_service().post([task] { (*task)(); });
|
||||
result.get();
|
||||
}
|
||||
void Remove(Handle h) {
|
||||
auto it = clients.find(h);
|
||||
if (it == clients.end())
|
||||
return;
|
||||
if (it->second.authorized)
|
||||
--viewers;
|
||||
outstanding -= it->second.bytes;
|
||||
clients.erase(it);
|
||||
}
|
||||
void Drop(Handle h) {
|
||||
// Cancel even a partially sent raw picture on revocation. Waiting for a
|
||||
// WebSocket close frame behind megabytes of video would keep it alive.
|
||||
Remove(h);
|
||||
websocketpp::lib::error_code ec;
|
||||
auto c = server.get_con_from_hdl(h, ec);
|
||||
if (!ec) {
|
||||
asio::error_code ignored;
|
||||
c->get_raw_socket().cancel(ignored);
|
||||
c->get_raw_socket().close(ignored);
|
||||
}
|
||||
}
|
||||
void Revoke(const std::string &view) {
|
||||
for (auto it = tickets.begin(); it != tickets.end();) {
|
||||
if (view.empty() || it->second.grant.view == view)
|
||||
it = tickets.erase(it);
|
||||
else
|
||||
++it;
|
||||
}
|
||||
std::vector<Handle> close;
|
||||
for (const auto &[h, c] : clients)
|
||||
if (view.empty() || c.view == view)
|
||||
close.push_back(h);
|
||||
for (auto h : close)
|
||||
Drop(h);
|
||||
}
|
||||
void RequestKey() {
|
||||
const auto now = Clock::now();
|
||||
if (now - last_key_request < 1s)
|
||||
return;
|
||||
last_key_request = now;
|
||||
key_requested = true;
|
||||
}
|
||||
void Message(Handle h, const Server::message_ptr &message) {
|
||||
auto it = clients.find(h);
|
||||
if (it == clients.end())
|
||||
return;
|
||||
auto &c = it->second;
|
||||
auto j = json::parse(message->get_payload(), nullptr, false);
|
||||
if (message->get_opcode() != websocketpp::frame::opcode::TEXT ||
|
||||
!j.is_object())
|
||||
return Drop(h);
|
||||
if (!c.authorized) {
|
||||
if (j.size() != 1 || !j.contains("ticket") || !j["ticket"].is_string())
|
||||
return Drop(h);
|
||||
auto ticket = tickets.find(j["ticket"].get<std::string>());
|
||||
if (ticket == tickets.end())
|
||||
return Drop(h);
|
||||
const auto grant = std::move(ticket->second);
|
||||
tickets.erase(ticket);
|
||||
if (!connected || grant.expires <= Clock::now() || viewers >= 8 ||
|
||||
(c.origin != grant.grant.origin &&
|
||||
!(grant.grant.apps && AppOrigin(c.origin))))
|
||||
return Drop(h);
|
||||
c.authorized = true;
|
||||
c.view = grant.grant.view;
|
||||
c.view_generation = grant.grant.view_generation;
|
||||
c.session = grant.grant.session;
|
||||
++viewers;
|
||||
// Also prompts a raw viewer to receive the current fresh picture.
|
||||
key_requested = true;
|
||||
return;
|
||||
}
|
||||
if (j.value("type", "") == "keyframe" && j.size() == 1 && !decoded) {
|
||||
c.waiting_key = true;
|
||||
RequestKey();
|
||||
return;
|
||||
}
|
||||
if (j.value("type", "") != "ack" || j.size() != 3 ||
|
||||
!j.contains("generation") || !j["generation"].is_number_unsigned() ||
|
||||
!j.contains("sequence") || !j["sequence"].is_number_unsigned() ||
|
||||
c.receipts.empty() ||
|
||||
j["generation"] != c.receipts.front().generation ||
|
||||
j["sequence"] != c.receipts.front().sequence)
|
||||
return Drop(h);
|
||||
c.bytes -= c.receipts.front().bytes;
|
||||
outstanding -= c.receipts.front().bytes;
|
||||
c.receipts.pop_front();
|
||||
}
|
||||
void Maintain() {
|
||||
timer = server.set_timer(
|
||||
250, [this](const websocketpp::lib::error_code &error) {
|
||||
if (error)
|
||||
return;
|
||||
const auto now = Clock::now();
|
||||
for (auto it = tickets.begin(); it != tickets.end();)
|
||||
if (it->second.expires <= now)
|
||||
it = tickets.erase(it);
|
||||
else
|
||||
++it;
|
||||
std::vector<Handle> close;
|
||||
for (auto &[h, c] : clients) {
|
||||
if ((!c.authorized && now - c.opened >= 3s) ||
|
||||
(!c.receipts.empty() && now - c.receipts.front().sent >= 10s)) {
|
||||
close.push_back(h);
|
||||
} else if (c.authorized && now - c.pinged >= 10s) {
|
||||
c.pinged = now;
|
||||
websocketpp::lib::error_code ec;
|
||||
server.ping(h, "", ec);
|
||||
if (ec)
|
||||
close.push_back(h);
|
||||
}
|
||||
}
|
||||
for (auto h : close)
|
||||
Drop(h);
|
||||
Maintain();
|
||||
});
|
||||
}
|
||||
void Enqueue(Packet p) {
|
||||
std::lock_guard lock(mutex);
|
||||
if (p.raw) {
|
||||
dropped += queue.size();
|
||||
queue.clear();
|
||||
queued_bytes = 0;
|
||||
} else {
|
||||
if (queue.size() >= 60 || queued_bytes + p.Size() > kMaxQueued) {
|
||||
dropped += queue.size();
|
||||
queue.clear();
|
||||
queued_bytes = 0;
|
||||
wait_key = true;
|
||||
key_requested = true;
|
||||
}
|
||||
if (wait_key && !p.Key()) {
|
||||
++dropped;
|
||||
return;
|
||||
}
|
||||
wait_key = false;
|
||||
}
|
||||
queued_bytes += p.Size();
|
||||
queue.push_back(std::move(p));
|
||||
if (!scheduled) {
|
||||
scheduled = true;
|
||||
server.get_io_service().post([this] { Drain(); });
|
||||
}
|
||||
}
|
||||
void Drain() {
|
||||
Packet p;
|
||||
{
|
||||
std::lock_guard lock(mutex);
|
||||
if (queue.empty()) {
|
||||
scheduled = false;
|
||||
return;
|
||||
}
|
||||
p = std::move(queue.front());
|
||||
queued_bytes -= p.Size();
|
||||
queue.pop_front();
|
||||
}
|
||||
if (connected && p.Generation() == generation && bool(p.raw) == decoded)
|
||||
Publish(p);
|
||||
// Let authorization, reads, and timers run between large pictures.
|
||||
server.get_io_service().post([this] { Drain(); });
|
||||
}
|
||||
void Publish(const Packet &p) {
|
||||
std::vector<Handle> ready;
|
||||
for (auto &[h, c] : clients) {
|
||||
if (!c.authorized)
|
||||
continue;
|
||||
// ACKs are flow-control hints, not proof that a client read the socket.
|
||||
// Also bound the actual send queue against forged early receipts.
|
||||
const auto buffered = server.get_con_from_hdl(h)->get_buffered_amount();
|
||||
if ((decoded && (!c.receipts.empty() || buffered > 0)) ||
|
||||
(!decoded && (c.receipts.size() >= 8 || c.bytes >= 1024 * 1024 ||
|
||||
buffered >= 1024 * 1024))) {
|
||||
++dropped;
|
||||
c.waiting_key = true;
|
||||
if (!decoded)
|
||||
RequestKey();
|
||||
continue;
|
||||
}
|
||||
if (!decoded && c.waiting_key && !p.Key()) {
|
||||
RequestKey();
|
||||
continue;
|
||||
}
|
||||
ready.push_back(h);
|
||||
}
|
||||
if (ready.empty())
|
||||
return;
|
||||
// Prepare once and share the immutable WebSocket message across viewers.
|
||||
// No length-prefixed pipe, Node Buffer, or per-viewer payload copy.
|
||||
auto message =
|
||||
server.get_con_from_hdl(ready.front())
|
||||
->get_message(websocketpp::frame::opcode::BINARY, p.Size());
|
||||
auto &bytes = message->get_raw_payload();
|
||||
Number(bytes, decoded ? 0x43444e31 : 0x43445631, 4);
|
||||
Number(bytes, p.Generation(), 4);
|
||||
Number(bytes, p.Sequence(), 4);
|
||||
const auto time =
|
||||
p.raw ? std::chrono::duration_cast<std::chrono::microseconds>(
|
||||
p.raw->queued.time_since_epoch())
|
||||
.count()
|
||||
: p.encoded.timestamp_us;
|
||||
Number(bytes, time, 8);
|
||||
Number(bytes, p.Key() ? 1 : 0, 4);
|
||||
if (p.raw) {
|
||||
Number(bytes, p.raw->width, 4);
|
||||
Number(bytes, p.raw->height, 4);
|
||||
message->append_payload(p.raw->nv12.data(), p.raw->nv12.size());
|
||||
} else
|
||||
message->append_payload(p.encoded.data.data(), p.encoded.data.size());
|
||||
message->set_header(websocketpp::frame::prepare_header(
|
||||
websocketpp::frame::basic_header(websocketpp::frame::opcode::BINARY,
|
||||
bytes.size(), true, false),
|
||||
websocketpp::frame::extended_header(bytes.size())));
|
||||
message->set_prepared(true);
|
||||
for (auto h : ready) {
|
||||
auto &c = clients.at(h);
|
||||
websocketpp::lib::error_code ec;
|
||||
server.send(h, message, ec);
|
||||
if (ec) {
|
||||
Drop(h);
|
||||
continue;
|
||||
}
|
||||
c.waiting_key = false;
|
||||
c.bytes += bytes.size();
|
||||
outstanding += bytes.size();
|
||||
c.receipts.push_back({uint32_t(p.Generation()), uint32_t(p.Sequence()),
|
||||
bytes.size(), Clock::now()});
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
VideoServer::VideoServer() : state_(std::make_unique<State>()) {}
|
||||
VideoServer::~VideoServer() = default;
|
||||
std::string VideoServer::Url() const {
|
||||
return "ws://" + state_->authority + "/api/video";
|
||||
}
|
||||
void VideoServer::Bind(uint64_t generation, bool decoded, bool connected) {
|
||||
auto &s = *state_;
|
||||
// Only the automation owner calls Bind; the I/O thread reads these values.
|
||||
s.Call([&s, generation, decoded, connected] {
|
||||
if (s.generation == generation && s.decoded == decoded &&
|
||||
s.connected == connected)
|
||||
return;
|
||||
s.Revoke("");
|
||||
s.generation = generation;
|
||||
s.decoded = decoded;
|
||||
s.connected = connected;
|
||||
s.key_requested = false;
|
||||
std::lock_guard lock(s.mutex);
|
||||
s.queue.clear();
|
||||
s.queued_bytes = 0;
|
||||
s.wait_key = true;
|
||||
});
|
||||
}
|
||||
void VideoServer::Grant(VideoGrant grant) {
|
||||
if (!Token(grant.ticket) || !Token(grant.view) || !Token(grant.session) ||
|
||||
!grant.view_generation || !BrowserOrigin(grant.origin))
|
||||
throw std::invalid_argument("INVALID_VIDEO_GRANT");
|
||||
auto &s = *state_;
|
||||
s.Call([&s, grant = std::move(grant)] {
|
||||
if (!s.connected)
|
||||
throw std::invalid_argument("NOT_READY");
|
||||
for (auto it = s.tickets.begin(); it != s.tickets.end();)
|
||||
if (it->second.expires <= Clock::now())
|
||||
it = s.tickets.erase(it);
|
||||
else
|
||||
++it;
|
||||
if (s.tickets.size() >= 32)
|
||||
throw std::invalid_argument("VIDEO_LIMIT");
|
||||
// A replaced view cannot retain a socket or consume an old grant.
|
||||
bool changed = false;
|
||||
for (const auto &[h, c] : s.clients)
|
||||
changed |=
|
||||
c.view == grant.view && (c.view_generation != grant.view_generation ||
|
||||
c.session != grant.session);
|
||||
for (const auto &[ticket, t] : s.tickets)
|
||||
changed |= t.grant.view == grant.view &&
|
||||
(t.grant.view_generation != grant.view_generation ||
|
||||
t.grant.session != grant.session);
|
||||
if (changed)
|
||||
s.Revoke(grant.view);
|
||||
if (!s.tickets
|
||||
.emplace(grant.ticket, State::Ticket{grant, Clock::now() + 15s})
|
||||
.second)
|
||||
throw std::invalid_argument("INVALID_VIDEO_GRANT");
|
||||
});
|
||||
}
|
||||
void VideoServer::Revoke(const std::string &view) {
|
||||
state_->Call([&] { state_->Revoke(view); });
|
||||
}
|
||||
bool VideoServer::Enabled() const { return state_->viewers.load() > 0; }
|
||||
bool VideoServer::TakeKeyframeRequest() {
|
||||
return state_->key_requested.exchange(false);
|
||||
}
|
||||
void VideoServer::Submit(std::shared_ptr<const DecodedFrame> frame) {
|
||||
if (!frame || !frame->width || !frame->height || (frame->width & 1) ||
|
||||
(frame->height & 1) || frame->width > 8192 || frame->height > 8192 ||
|
||||
uint64_t(frame->width) * frame->height > 16777216 ||
|
||||
frame->nv12.size() != size_t(frame->width) * frame->height * 3 / 2)
|
||||
throw std::invalid_argument("INVALID_VIDEO_FRAME");
|
||||
if (Enabled())
|
||||
state_->Enqueue({std::move(frame), {}});
|
||||
}
|
||||
void VideoServer::Submit(EncodedVideoFrame frame) {
|
||||
if (frame.data.empty() || frame.data.size() > kMaxEncoded)
|
||||
throw std::invalid_argument("INVALID_VIDEO_FRAME");
|
||||
if (Enabled())
|
||||
state_->Enqueue({{}, std::move(frame)});
|
||||
}
|
||||
json VideoServer::Status() const {
|
||||
const auto &s = *state_;
|
||||
std::lock_guard lock(s.mutex);
|
||||
return {{"transport", "loopback-websocket/1"},
|
||||
{"viewers", s.viewers.load()},
|
||||
{"queued", s.queue.size()},
|
||||
{"queued_bytes", s.queued_bytes},
|
||||
{"unacknowledged_bytes", s.outstanding.load()},
|
||||
{"dropped", s.dropped.load()}};
|
||||
}
|
||||
} // namespace crossdesk::cli
|
||||
@@ -0,0 +1,48 @@
|
||||
/*
|
||||
* @Author: DI JUNKUN
|
||||
* @Date: 2026-11-03
|
||||
* Copyright (c) 2026 by DI JUNKUN, All Rights Reserved.
|
||||
*/
|
||||
|
||||
#ifndef _VIDEO_SERVER_H_
|
||||
#define _VIDEO_SERVER_H_
|
||||
|
||||
#include <controller_session.h>
|
||||
#include <memory>
|
||||
#include <string>
|
||||
|
||||
namespace crossdesk::cli {
|
||||
|
||||
struct VideoGrant {
|
||||
std::string ticket, view, session, origin;
|
||||
uint64_t view_generation = 0;
|
||||
bool apps = false;
|
||||
};
|
||||
|
||||
// Owns the loopback listener and its I/O thread. JSON-RPC carries only grants
|
||||
// and diagnostics; encoded and decoded pictures stay in the native process.
|
||||
class VideoServer {
|
||||
public:
|
||||
VideoServer();
|
||||
~VideoServer();
|
||||
VideoServer(const VideoServer &) = delete;
|
||||
VideoServer &operator=(const VideoServer &) = delete;
|
||||
|
||||
std::string Url() const;
|
||||
void Bind(uint64_t generation, bool decoded, bool connected);
|
||||
void Grant(VideoGrant grant);
|
||||
void Revoke(const std::string &view = "");
|
||||
bool Enabled() const;
|
||||
bool TakeKeyframeRequest();
|
||||
void Submit(std::shared_ptr<const DecodedFrame> frame);
|
||||
void Submit(EncodedVideoFrame frame);
|
||||
nlohmann::json Status() const;
|
||||
|
||||
private:
|
||||
struct State;
|
||||
std::unique_ptr<State> state_;
|
||||
};
|
||||
|
||||
} // namespace crossdesk::cli
|
||||
|
||||
#endif
|
||||
+7
-3
@@ -4,6 +4,8 @@ local terminal_dir = path.join(source_dir, "libs/terminal")
|
||||
local minirtc_dir = path.join(source_dir, "deps/submodules/minirtc")
|
||||
|
||||
add_requires("spdlog 1.14.1", {system = false})
|
||||
add_requires("asio 1.32.0", "websocketpp 0.8.2", {system = false})
|
||||
add_requires("libpng v1.6.59", {system = false, configs = {shared = false}})
|
||||
if is_plat("windows") then
|
||||
add_requires("zlib 1.3.1", {system = false, configs = {shared = false}})
|
||||
end
|
||||
@@ -11,7 +13,7 @@ end
|
||||
target("crossdesk_controller")
|
||||
set_kind("static")
|
||||
add_deps("minirtc", "crossdesk_wire")
|
||||
add_packages("spdlog")
|
||||
add_packages("spdlog", "libpng", "libyuv")
|
||||
add_includedirs(path.join(controller_dir, "include"),
|
||||
path.join(minirtc_dir, "src/api"), {public = true})
|
||||
add_includedirs(path.join(minirtc_dir, "src/log"))
|
||||
@@ -32,12 +34,14 @@ target_end()
|
||||
|
||||
target("crossdesk-cli")
|
||||
set_kind("binary")
|
||||
set_version("0.1.5")
|
||||
set_version("0.2.0")
|
||||
add_deps("crossdesk_terminal")
|
||||
add_files("main.cpp")
|
||||
add_files("agent.cpp", "video_server.cpp")
|
||||
add_packages("asio", "websocketpp")
|
||||
add_files(is_plat("windows") and "terminal_io_windows.cpp" or "terminal_io_posix.cpp")
|
||||
if is_plat("windows") then
|
||||
add_syslinks("advapi32", "user32")
|
||||
add_syslinks("advapi32", "user32", "ws2_32")
|
||||
add_defines("NOMINMAX", "WIN32_LEAN_AND_MEAN")
|
||||
elseif is_plat("linux") then
|
||||
add_syslinks("rt")
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
dist/
|
||||
@@ -0,0 +1,198 @@
|
||||
# CrossDesk MCP
|
||||
|
||||
让本机 Agent 通过 MCP 观察并操作远端 CrossDesk，同时为用户提供 MCP Apps 或浏览器监看面板。控制端不需要安装 CrossDesk 桌面客户端；被控端仍需运行兼容的 CrossDesk，并连接支持 `cli` 注册的信令服务器。
|
||||
|
||||
此目录实现 MCP 包 `@chunkdash/crossdesk-mcp` 0.1.0，绑定原生 CLI 包 0.2.0。**源码实现和本地打包不代表这些版本已经发布到 npm。** 发布后的安装示例需要以实际可用版本为准。
|
||||
|
||||
## 从源码运行
|
||||
|
||||
仓库根目录构建原生程序（已有工作区可保留原有桌面构建选项）：
|
||||
|
||||
```sh
|
||||
sh apps/cli/scripts/prepare-deps.sh
|
||||
xmake f -y --CROSSDESK_BUILD_DESKTOP=n --CROSSDESK_BUILD_CLI=y --USE_CUDA=n -m release
|
||||
xmake build crossdesk-cli
|
||||
cd apps/mcp
|
||||
npm ci
|
||||
npm run build
|
||||
```
|
||||
|
||||
Node.js 要求 22 或以上。在 Agent 宿主中配置本地 stdio 进程，例如 macOS arm64 源码构建：
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"crossdesk": {
|
||||
"command": "node",
|
||||
"args": [
|
||||
"/absolute/path/crossdesk/apps/mcp/dist/main.js",
|
||||
"serve", "--stdio", "--panel=auto",
|
||||
"--native-binary", "/absolute/path/crossdesk/build/macosx/arm64/release/crossdesk-cli"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Windows 和 Linux 使用各自构建产物路径；`--native-binary` 必须指向可信原生可执行程序，不接受 shell 命令。也可以通过 `CROSSDESK_NATIVE_BINARY` 显式指定开发产物。未指定时只从包内的固定版本平台依赖解析，不搜索 PATH。服务启动不连接远端、不打开浏览器；连接时才检查 `doctor` 协议并启动原生子进程。
|
||||
|
||||
发布包的配置形式：
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"crossdesk": {
|
||||
"command": "npx",
|
||||
"args": ["-y", "@chunkdash/crossdesk-mcp@<已发布版本>", "serve", "--stdio", "--panel=auto"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
平台依赖覆盖 macOS arm64/x64、glibc Linux arm64/x64 和 Windows x64。安装时保留 optional dependencies；不需要另装全局 CLI。原生运行要求沿用 [CLI 分发文档](../cli/docs/npm-distribution.md)。`npm ci` 仅安装此源码目录的开发依赖，开发运行需使用 `--native-binary`；生成的发布包才包含固定平台依赖。
|
||||
|
||||
## 用户连接流程
|
||||
|
||||
1. 对 Agent 说“打开 CrossDesk”。`crossdesk_open` 创建 `view_id`，返回面板选择和等待用户状态。
|
||||
2. 在面板选择已保存设备，或输入 9 位设备 ID；密码填写在面板里。可选择仅查看或允许 Agent 控制。
|
||||
3. “保存设备”“记住密码”“允许以后自动连接”分别控制设备簿、系统凭据库、持久权限。后两项要求保存设备，默认均关闭。保存密码不自动授予控制权限。
|
||||
4. 有待执行的远程任务时，Agent 在打开面板后立即调用 `crossdesk_get_session({view_id, wait_ms: 25000})` 等待连接。连接就绪后，工具直接返回会话 ID 和权限状态，Agent 继续任务；用户无需回到对话发送连接确认，也无需点击继续按钮。浏览器和 Apps 面板使用同一流程。
|
||||
5. Agent 获取独立截图，用截图的 `observation_id` 和图像坐标提交动作。面板预览不消耗模型截图序列。
|
||||
|
||||
`crossdesk_open` 立即返回，让 Apps 宿主有机会渲染面板；等待连接在后续 `crossdesk_get_session` 调用中完成。按 `view_id` 查询默认最多等待 25 秒，`wait_ms: 0` 则立即查询；未就绪时返回 `wait_timed_out: true`，Agent 在原任务仍待执行时继续等待。按 `session_id` 查询仍立即返回。取消等待只取消本次查询，不断开用户正在建立或使用的连接。用户主动断开后返回 `closed`，Agent 停止等待，不自动重连。
|
||||
|
||||
自动继续依赖宿主保持当前 Agent 任务及工具调用流程。它不通过网页伪造用户消息，也不承诺从已经结束的对话中主动唤醒 Agent；此类唤醒由宿主负责。
|
||||
|
||||
人工暂停由原生层锁定，模型不能通过 `set_control(true)` 恢复；只有当前授权面板的恢复按钮可以恢复。仅查看连接不能升级为控制连接，需重新连接并授权。关闭/隐藏面板仅停止预览，远程会话保留；断开按钮或 `crossdesk_disconnect` 才结束会话。
|
||||
|
||||
同一 MCP 进程默认只保留一个活动远程会话。相同配置目录中的进程用设备锁避免同时连接同一目标；它不协调另一个系统用户、另一份配置目录或独立桌面客户端。
|
||||
|
||||
## 面板策略
|
||||
|
||||
`--panel` 优先于设备配置中的 `panel`，默认 `auto`。
|
||||
|
||||
| 模式 | 行为 |
|
||||
| --- | --- |
|
||||
| `auto` | 当前请求声明 MCP Apps MIME 能力时提供 Apps；其他情况打开本机浏览器 |
|
||||
| `apps` | 只使用 MCP Apps；能力不匹配返回 `PANEL_UNSUPPORTED` |
|
||||
| `browser` | 只使用本地浏览器，不附加 Apps 工具资源元数据 |
|
||||
| `off` | 不启动网页服务，保留截图和操作工具，凭据通过独立 `setup` 配置 |
|
||||
|
||||
使用 `extensions["io.modelcontextprotocol/ui"].mimeTypes` 中的 `text/html;profile=mcp-app` 判断能力。旧协议读取 initialize 能力；2026-07-28 协议读取当前请求的能力，不按宿主产品名猜测支持情况。Apps 的卡片、侧栏或窗口形式由宿主决定。
|
||||
|
||||
Apps 资源为 `ui://crossdesk/viewer.html`，工具只在允许使用 Apps 且当前请求支持时暴露 UI 元数据和面板专用接口。页面回报 `crossdesk_ui_ready` 后才标记就绪；10 秒未回报显示 `not_confirmed`，不自动弹第二个浏览器。用户明确要求回退时调用 `crossdesk_open_browser`；该工具不附带 Apps 元数据，`apps`/`off` 策略拒绝覆盖。
|
||||
|
||||
浏览器面板只监听随机端口的 `127.0.0.1`，与 MCP 服务同一进程。它不是 MCP Streamable HTTP，也不向局域网提供 MCP 接入。MCP、CLI、用户浏览器应在同一控制端电脑上运行；SSH 容器/远程宿主的浏览器转发不属于此本地接入模式。
|
||||
|
||||
页面标题为 CrossDesk MCP，仅保留正式应用标志，使用文字按钮和简洁状态提示。布局填满浏览器或宿主面板的可用宽高，远程画面保持原比例自动缩放；连接设置在高度不足时独立滚动。连接后侧栏显示设备与授权信息，暂停、恢复和断开按钮位于画面工具栏。「放大画面」收起侧栏，按 Esc 还原；窄屏默认显示连接表单，连接后显示画面，可用「连接信息」展开设备详情。界面资源均可离线使用，保存设备与持久授权仍由用户分别选择。
|
||||
|
||||
首次打开使用 URL fragment 中的短期一次性 bootstrap，页面立即清除 fragment，并换取 HttpOnly/SameSite cookie 和 CSRF 令牌。模型只收到无特权的网页地址。复制无凭据地址到另一浏览器不能获得会话控制权。
|
||||
|
||||
无法自动打开浏览器时，在自己的终端运行工具结果 `pairing_command` 中的命令和参数。例如：
|
||||
|
||||
```sh
|
||||
crossdesk-mcp setup --open-panel --view <view_id> --config-dir <相同的配置目录>
|
||||
```
|
||||
|
||||
通过源码运行时用 `node /absolute/path/apps/mcp/dist/main.js` 替换 `crossdesk-mcp`。配对通过本用户配置目录中的受限运行描述符和本地 IPC 完成，不允许匿名 HTTP 请求签发控制凭证。`setup` 使用用户终端的桌面环境打开浏览器；仍无法打开时，只在该交互终端显示 60 秒有效的一次性地址，不将它返回 MCP 或发送到聊天。面板最多保留 32 个视图；再次打开时会清理超过 30 分钟未使用的视图。浏览器会话最长 12 小时，失效后重新打开。
|
||||
|
||||
## 无面板配置与凭据
|
||||
|
||||
在用户自己的交互终端执行（不能在宿主 stdio 通道中交互输入密码）：
|
||||
|
||||
```sh
|
||||
crossdesk-mcp setup
|
||||
```
|
||||
|
||||
配置默认存放于 `~/.config/crossdesk/mcp/`，可用 `--config-dir` 或 `CROSSDESK_MCP_CONFIG_DIR` 覆盖。`devices.json` 包含设备、面板/视频选项及凭据引用；密码保存到 macOS Keychain、Windows 凭据库或 Linux Secret Service。凭据库不可用就返回明确错误，没有明文文件降级。面板的临时连接仍可使用用户现场输入的密码。持久保存失败会保留已建立连接，并在面板显示错误。
|
||||
|
||||
也可在 `setup` 中指定外部密码环境变量名称，服务读取该变量作为凭据提供器。示例配置（没有密码值）：
|
||||
|
||||
```json
|
||||
{
|
||||
"version": 1,
|
||||
"panel": "off",
|
||||
"server": "api.crossdesk.cn",
|
||||
"port": 9099,
|
||||
"devices": [{
|
||||
"alias": "work-pc",
|
||||
"device_id": "123456789",
|
||||
"server": "api.crossdesk.cn",
|
||||
"port": 9099,
|
||||
"authorization": "control",
|
||||
"credential": "environment",
|
||||
"environment_variable": "MY_CROSSDESK_PASSWORD"
|
||||
}]
|
||||
}
|
||||
```
|
||||
|
||||
授权范围和可用凭据缺一不可。模型不能提供任意 `device_id` 绕过面板确认；模型侧 `crossdesk_connect` 仅接受已授权设备别名。不要把密码发给 Agent，也不要把包含密码的命令行写进宿主配置。
|
||||
|
||||
Apps 依赖宿主遵守 app-only 工具可见性和 `_meta` 的模型不可见约定；服务另检查每个视图的随机 UI 能力令牌和 generation。app-only 标记本身不是可信身份凭据。能够读取同用户进程/文件或执行任意本机代码的 Agent 不受此边界隔离。
|
||||
|
||||
## 模型工具
|
||||
|
||||
| 工具 | 作用 |
|
||||
| --- | --- |
|
||||
| `crossdesk_open` | 打开设备选择/连接面板，返回 `view_id` |
|
||||
| `crossdesk_open_browser` | 用户显式请求的浏览器入口/回退 |
|
||||
| `crossdesk_list_devices` | 本机设备簿和授权摘要，不提供云发现 |
|
||||
| `crossdesk_connect` | 通过别名连接已经授权且有凭据的设备 |
|
||||
| `crossdesk_get_session` | `session_id`、`view_id` 严格二选一；按 `view_id` 可等待连接，`wait_ms` 范围 0–25000 |
|
||||
| `crossdesk_screenshot` | 返回 PNG image content 和观测元数据 |
|
||||
| `crossdesk_action` | 结构化输入，可附带动作之后的截图 |
|
||||
| `crossdesk_set_control` | 在已授权范围内启停 Agent 输入 |
|
||||
| `crossdesk_select_display` | 切换显示器，使旧观测失效 |
|
||||
| `crossdesk_disconnect` | 断开并释放输入状态、子进程和设备锁 |
|
||||
|
||||
动作包括 `move`、`click`、`double_click`、`scroll`、`drag`、`key_press`、`hotkey`、`type_text`。不提供 shell、任意键按下保持、剪贴板或文件操作。按键使用 Windows VK 对应的标准名称：`A`–`Z`、`0`–`9`、`F1`–`F12`、`CTRL`、`ALT`、`SHIFT`、`META`、`ENTER`、`ESC`、方向键等；应用快捷键仍需根据远端系统选择。
|
||||
|
||||
截图默认最长边 1280，最大 2048；允许指定源帧像素区域。返回 `source_size`、`source_rect`、`image_size`、`generation`、`geometry_epoch`、`frame_id`、`frame_age_ms` 和 `observation_id`。动作坐标使用返回图像坐标，不使用浏览器 CSS 像素。观测保留最多 32 个，30 秒过期；切换显示器、连接代际或图像几何变化立即失效。截图使用不超过 2 秒的帧，PNG 不超过 2 MiB，必要时继续缩小输出并返回真实尺寸。
|
||||
|
||||
`operation_id` 在当前原生会话中缓存最近 128 个操作，至少在未被容量淘汰的 5 分钟窗口内支持同参数查询；不同参数复用 ID 返回冲突。`submitted` 仅说明本地发送完成，远端没有执行 ACK。超时、断网、取消或发送失败可能返回 `outcome_unknown`/`side_effect_possible`，禁止自动重放。`observe_after` 默认开启，后续图像只证明取得更新帧，不证明操作执行成功。
|
||||
|
||||
## 实时视频与按需截图
|
||||
|
||||
实时视频提供两条路径，默认 `browser`，两条路径均由原生 CLI 的本机 WebSocket 直接发送给浏览器。Node 只处理会话、授权票据和 MCP 工具，不读取或转发实时视频字节；不再创建视频管道，不经过 MCP stdio 或模型上下文，不逐帧转 PNG、不重新编码视频。
|
||||
|
||||
| `video.decode_path` | 画面路径 |
|
||||
| --- | --- |
|
||||
| `browser` | MiniRTC 输出原始 H.264 Annex B → 浏览器 WebCodecs 解码 → canvas |
|
||||
| `minirtc` | MiniRTC 解码 → 紧凑 NV12 原始帧 → 浏览器 VideoFrame / canvas；浏览器不再解码 H.264 |
|
||||
|
||||
`video.decoder` 默认 `auto`：优先使用可用硬件解码器，不支持时使用软件。`software` 禁用本程序的硬件选择与硬件回退。浏览器路径使用 WebCodecs 的 `prefer-hardware` / `prefer-software`；这些值是浏览器 API 的偏好，浏览器可以忽略，不能据此宣称已确认物理硬件或软件后端。原生路径由 MiniRTC 的解码器工厂选择，本机 macOS 使用 VideoToolbox 或 OpenH264；其他平台使用 MiniRTC 已支持的后端。状态中的 `native_video_decoder` 返回 MiniRTC 报告的后端名称（统计可用前为空），不代表浏览器后端。
|
||||
|
||||
在 `~/.config/crossdesk/mcp/devices.json` 的根对象中加入：
|
||||
|
||||
```json
|
||||
"video": {
|
||||
"decode_path": "minirtc",
|
||||
"decoder": "auto"
|
||||
}
|
||||
```
|
||||
|
||||
也可在 Agent 宿主的 MCP 启动参数中配置，命令行优先于文件：
|
||||
|
||||
```sh
|
||||
crossdesk-mcp serve --stdio --video-path browser --video-decoder auto
|
||||
crossdesk-mcp serve --stdio --video-path minirtc --video-decoder software
|
||||
```
|
||||
|
||||
修改后重启 MCP，再建立连接。选项适用于整个 MCP 进程，两条路径均将解码选项传给原生端；浏览器路径中的原生解码用于截图缓存。截图工具或动作后的截图请求才触发 PNG 编码。这里配置的是控制端解码器，不更改远端编码器。原有配置文件无需新增字段即可继续使用。
|
||||
|
||||
Agent 模式默认请求高帧率档位，分辨率上限 1920 × 1080；实际分辨率和帧率由远端能力、网络和本机解码性能决定。面板显示实际尺寸与呈现 FPS。浏览器路径需要 WebCodecs H.264 解码能力；MiniRTC 路径仅需浏览器支持从 NV12 创建 VideoFrame 并绘制。能力不足时明确报错，不自动退回 PNG 轮询。NV12 不压缩，1080p / 30 FPS 约需 93 MB/s 本机传输，4K / 30 FPS 约需 373 MB/s；它不增加远端到控制端的 H.264 网络码率。移除 Node 转发没有消除 NV12 自身的本机传输与浏览器像素处理开销；这不是 GPU 零拷贝。
|
||||
|
||||
视频 WebSocket 在原生 CLI 的独立 I/O 线程中运行，只监听随机 `127.0.0.1` 端口，与 Node 面板 HTTP 端口不同。浏览器面板通过 cookie/CSRF 授权获取视频票据，Apps 通过 app-only `crossdesk_video_open` 和视图令牌获取。票据 15 秒内一次性有效，绑定视图、会话和代际，通过 WebSocket 第一条消息提交，不写入 URL、日志或模型可见工具结果。WebSocket 只提供画面、收帧 ACK 和关键帧恢复请求，不能执行远端输入操作。最多 8 个活动观看者。H.264 丢帧后等待关键帧恢复；NV12 每帧独立，每个观看者最多一帧未确认，原生发送队列只保留最新画面；H.264 最多 8 帧未确认并受字节阈值限制。10 秒未确认最早一帧则关闭该视频通道，不阻塞其他观看者或 MCP 控制。视图撤销、显示器/连接代际变化、断开和进程退出会关闭对应视频连接；暂停 Agent 控制仍继续播放。页面隐藏或卸载时释放视频订阅，恢复可见时重新订阅。
|
||||
|
||||
浏览器页面和 MCP Apps 资源的 CSP 允许原生服务使用的 `ws://127.0.0.1:*`（仅动态端口），HTTP 控制仍限制到 Node 面板的确切地址。原生服务另外校验确切 Host、Origin 和一次性票据。宿主仍可能限制 loopback、WebSocket、WebCodecs 或 iframe 权限；视频失败时提供重试和浏览器入口，`--panel=apps` 的严格策略仍禁止浏览器回退。浏览器和 Apps 共用同一播放器，具体宿主需要独立验收。
|
||||
|
||||
## 打包
|
||||
|
||||
```sh
|
||||
npm run build
|
||||
npm run pack:npm -- --out-dir /absolute/path/artifacts
|
||||
```
|
||||
|
||||
脚本生成可发布 tarball，包含面板、运行代码、源代码、许可和固定版本的五个平台可选依赖。CI 的 `MCP package` 工作流仅构建并上传 `mcp-package`，不发布 npm。原生包继续由 CLI 工作流构建，`doctor` 必须同时声明自动化协议和 `automation_video_protocols: ["h264-annexb/1", "nv12/1"]` 和 `automation_video_decoders: ["auto", "software"]`，以及 `automation_video_transport: "loopback-websocket/1"`；先发布对应 0.2.0 平台包，再发布 MCP tarball。尚未进行真实跨平台发布/远端验收时，不应把构建成功等同于各宿主和设备上的联调完成。
|
||||
|
||||
从发布 tarball 重建 TypeScript/面板时，将包中的 `build-package.json` 复制为 `package.json`，运行 `npm ci && npm run build`。原生源码和依赖来源随平台包提供，详见 [源码来源](../cli/docs/source-provenance.md)。
|
||||
Generated
+977
@@ -0,0 +1,977 @@
|
||||
{
|
||||
"name": "@chunkdash/crossdesk-mcp",
|
||||
"version": "0.1.0",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "@chunkdash/crossdesk-mcp",
|
||||
"version": "0.1.0",
|
||||
"license": "GPL-3.0-only",
|
||||
"dependencies": {
|
||||
"@modelcontextprotocol/client": "2.4.0",
|
||||
"@modelcontextprotocol/core": "2.4.0",
|
||||
"@modelcontextprotocol/ext-apps": "2.0.4",
|
||||
"@modelcontextprotocol/server": "2.4.0",
|
||||
"@napi-rs/keyring": "2.1.2",
|
||||
"zod": "4.6.7"
|
||||
},
|
||||
"bin": {
|
||||
"crossdesk-mcp": "dist/main.js"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "22.19.15",
|
||||
"esbuild": "0.28.2",
|
||||
"typescript": "5.9.3"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=22"
|
||||
}
|
||||
},
|
||||
"node_modules/@esbuild/aix-ppc64": {
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.28.2.tgz",
|
||||
"integrity": "sha512-XExcO+dvLKvVtNTibSTBej1NCAbaGhWn9Ww1ZPx80qsahhPFe/8jgWP0IchNe0F3HwkU7n8ejhH8bjonqht8mQ==",
|
||||
"cpu": [
|
||||
"ppc64"
|
||||
],
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"aix"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
}
|
||||
},
|
||||
"node_modules/@esbuild/android-arm": {
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/android-arm/-/android-arm-0.28.2.tgz",
|
||||
"integrity": "sha512-kXXoiPVVGQcnIYGOeaovwOURpniDBpSq4A03qkQ+BMQqtGG6HYap3xne9C1O1yo4TR3qxlCX5IqqmX6fFo2Lqg==",
|
||||
"cpu": [
|
||||
"arm"
|
||||
],
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"android"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
}
|
||||
},
|
||||
"node_modules/@esbuild/android-arm64": {
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/android-arm64/-/android-arm64-0.28.2.tgz",
|
||||
"integrity": "sha512-5YfKeeI8qWfBZIX+u2xZC3Zlb3Os/gLS2sbEKM+I4ZOcsWmHS2WLysCcQZDAFRslDUU5Oiq44gf6PYN1vGwG5A==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"android"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
}
|
||||
},
|
||||
"node_modules/@esbuild/android-x64": {
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/android-x64/-/android-x64-0.28.2.tgz",
|
||||
"integrity": "sha512-O387ite7SzUyCcy3JQX4P4bLtEA7bLLkx+esve5JHnyYfNTxcVpXZo9jhdB0lTKN44gztELTdU7nS8Nr16Fs1Q==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"android"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
}
|
||||
},
|
||||
"node_modules/@esbuild/darwin-arm64": {
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/darwin-arm64/-/darwin-arm64-0.28.2.tgz",
|
||||
"integrity": "sha512-n4KqkOQrraxHJcgjM1RvwbigfQKIKJVpM7xp+KsxiyUSrRdIXnt73VhrPAx0fV44hgfmIVKjxMN9J1t5jySVkw==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"darwin"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
}
|
||||
},
|
||||
"node_modules/@esbuild/darwin-x64": {
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/darwin-x64/-/darwin-x64-0.28.2.tgz",
|
||||
"integrity": "sha512-uq6suIWYP37qzGddBKPw5QEQPi6HiLGsO7UmkpfyaYNQ3D+rN6w6WfwH+nuqcGXWvawGwxOEroO4YGnFh95azw==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"darwin"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
}
|
||||
},
|
||||
"node_modules/@esbuild/freebsd-arm64": {
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/freebsd-arm64/-/freebsd-arm64-0.28.2.tgz",
|
||||
"integrity": "sha512-n+I0BTSRIoy+d6RPKnEVwql5UwBJolytvY4mAOIEJorKlqgPII8ix6slVVrfZ5Tnj7glIZvloylbB/EJPMWEXw==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"freebsd"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
}
|
||||
},
|
||||
"node_modules/@esbuild/freebsd-x64": {
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/freebsd-x64/-/freebsd-x64-0.28.2.tgz",
|
||||
"integrity": "sha512-78XJTJkvPs0kz2w61301PJjXl4g7q3JqiYMZ/M/yVI73EHBrCRTgkhu9oqG7vPqq+a/yadEW8aD+agKlk5xrmg==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"freebsd"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
}
|
||||
},
|
||||
"node_modules/@esbuild/linux-arm": {
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/linux-arm/-/linux-arm-0.28.2.tgz",
|
||||
"integrity": "sha512-XlDnu2q5yoqems+xay6wSAcg9DDD7K9RLKZEBOMZm3ckNpJBvOX20tSfby8KfrrhINDyv9V2YVZKY/SpoGJI8w==",
|
||||
"cpu": [
|
||||
"arm"
|
||||
],
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
}
|
||||
},
|
||||
"node_modules/@esbuild/linux-arm64": {
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/linux-arm64/-/linux-arm64-0.28.2.tgz",
|
||||
"integrity": "sha512-pW4AC0P3it8c7do9MVM4p51FzHzdM/TZrerurgRcHJ2WTa1VQ1CIq18xncfpBJw4ojkiZZrKW2yIBWBP92j6Ug==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
}
|
||||
},
|
||||
"node_modules/@esbuild/linux-ia32": {
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/linux-ia32/-/linux-ia32-0.28.2.tgz",
|
||||
"integrity": "sha512-CYbnj78HsIeA+DhgUKgFCfvNsTHFhMMrinUrMZpDXJXKN8T3XViTZ/+wtHeVxEWY8ewSzTFN+nRmSwO2tZaLUQ==",
|
||||
"cpu": [
|
||||
"ia32"
|
||||
],
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
}
|
||||
},
|
||||
"node_modules/@esbuild/linux-loong64": {
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/linux-loong64/-/linux-loong64-0.28.2.tgz",
|
||||
"integrity": "sha512-buwkd8nsph4R+ajRvw0qM5Hja/TXQow3ptzWO2EbG/cqcIkHloRrdlBtQlshyYGTNFvfkfJ5tpPLVkY4DtsPfQ==",
|
||||
"cpu": [
|
||||
"loong64"
|
||||
],
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
}
|
||||
},
|
||||
"node_modules/@esbuild/linux-mips64el": {
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/linux-mips64el/-/linux-mips64el-0.28.2.tgz",
|
||||
"integrity": "sha512-ZVykbDyk7519VwiNb9Lcj9m8XM6v5V9uKPvrEMkkEedVewf+0itkhahp4HDpgERXhwLRpWFypsGbG/J8s0QjJA==",
|
||||
"cpu": [
|
||||
"mips64el"
|
||||
],
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
}
|
||||
},
|
||||
"node_modules/@esbuild/linux-ppc64": {
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/linux-ppc64/-/linux-ppc64-0.28.2.tgz",
|
||||
"integrity": "sha512-CAXl+Dtd9UUuJd8pKKdwh6MLm3MUMiqMPmhZ3tTSXPqfyQ3vDl6R5hZdZ/kYojK4ofXtdfSv1tFq8XzWx3heNQ==",
|
||||
"cpu": [
|
||||
"ppc64"
|
||||
],
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
}
|
||||
},
|
||||
"node_modules/@esbuild/linux-riscv64": {
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/linux-riscv64/-/linux-riscv64-0.28.2.tgz",
|
||||
"integrity": "sha512-GeXCej4IQtU1B+QlDV8W/RRvbzI3O/Stss+/bCXv4lZls5WGRtu2a+3JkA3i4qIUlMXpcHebWpF8AkJhATowuA==",
|
||||
"cpu": [
|
||||
"riscv64"
|
||||
],
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
}
|
||||
},
|
||||
"node_modules/@esbuild/linux-s390x": {
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/linux-s390x/-/linux-s390x-0.28.2.tgz",
|
||||
"integrity": "sha512-3H1weTYZPxt/WOhByszQZybS9w5lKzUn1FDMsgEChbHWQwHYQQRfBxgCcZvPhjHfKyJjIievvMmEUawJrdY9Dg==",
|
||||
"cpu": [
|
||||
"s390x"
|
||||
],
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
}
|
||||
},
|
||||
"node_modules/@esbuild/linux-x64": {
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/linux-x64/-/linux-x64-0.28.2.tgz",
|
||||
"integrity": "sha512-4xTZr1FUmSoQW4XIWmit3tzQrUTZM+N3P0XV8xROKYF50XfI7xeO90+1bZvNwxIufQ9hDQVRJH5YhgPVF8A/HQ==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
}
|
||||
},
|
||||
"node_modules/@esbuild/netbsd-arm64": {
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/netbsd-arm64/-/netbsd-arm64-0.28.2.tgz",
|
||||
"integrity": "sha512-sSATRjPeDBg3pdgHoQfoYBob11Kk1FGa9lui5RIHZCoCkJa9QKlvl3/vKz2usCmYYjs7ymJR/2Nnsqe+Hjt5nw==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"netbsd"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
}
|
||||
},
|
||||
"node_modules/@esbuild/netbsd-x64": {
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/netbsd-x64/-/netbsd-x64-0.28.2.tgz",
|
||||
"integrity": "sha512-lqnzCV+mM0gIADaKihiCg6ifgfU2L3h5E33rNQBN1Y4MaVGnzryzmvvf7UHxprpQdE8hpqLolJ9Rl+SkIRDpyw==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"netbsd"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
}
|
||||
},
|
||||
"node_modules/@esbuild/openbsd-arm64": {
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/openbsd-arm64/-/openbsd-arm64-0.28.2.tgz",
|
||||
"integrity": "sha512-AL2qJILH7lNjrDmCQDvdxMfAUIv8KMNZOvrwAQ8i8//ntL9FflhOyMJ8OZSMBb8/AWXe3/5v5S20y3zCoZWKoQ==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"openbsd"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
}
|
||||
},
|
||||
"node_modules/@esbuild/openbsd-x64": {
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/openbsd-x64/-/openbsd-x64-0.28.2.tgz",
|
||||
"integrity": "sha512-QtiuPytchRyC4rwUKhexJdQKvDuZ6hWloi3igqPQNUJCS1/v9EiO3UTOXR6A3FoMo4fnAKbWJdqaIwhOzh8qEw==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"openbsd"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
}
|
||||
},
|
||||
"node_modules/@esbuild/openharmony-arm64": {
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/openharmony-arm64/-/openharmony-arm64-0.28.2.tgz",
|
||||
"integrity": "sha512-WkhYDmpTjLvGlScA1rwjRUmhl4k8oXR3cIbtqWmELgU/dFeHHlEllxDvdWcNJV9rbzCexB5vz8gtNewWLgCT7Q==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"openharmony"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
}
|
||||
},
|
||||
"node_modules/@esbuild/sunos-x64": {
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/sunos-x64/-/sunos-x64-0.28.2.tgz",
|
||||
"integrity": "sha512-GPMSkTOtMnv2U2F8gxe4Io6qmVs+YKyp832Etqqxr0hFngmXQ3rzwytelm3GIn7T4VviRUlf3sOgBOiTdvaf7g==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"sunos"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
}
|
||||
},
|
||||
"node_modules/@esbuild/win32-arm64": {
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/win32-arm64/-/win32-arm64-0.28.2.tgz",
|
||||
"integrity": "sha512-PIhhEkE9uPBleRBrQEJpUn7MBnibZzbGzYWPmY3x+YoVg/95zbjB4CxPPOQ8l5tYYM4mMaCthF8/1DIfBQQyWQ==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"win32"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
}
|
||||
},
|
||||
"node_modules/@esbuild/win32-ia32": {
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/win32-ia32/-/win32-ia32-0.28.2.tgz",
|
||||
"integrity": "sha512-YmJbfTlvU7Sdn9BB+4PRES4oB6pxgS37MAONj+hBr/cpXS1aBPKXxNnDbu+QCWPj0o9dgyxeq79g6c5P8KeuYA==",
|
||||
"cpu": [
|
||||
"ia32"
|
||||
],
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"win32"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
}
|
||||
},
|
||||
"node_modules/@esbuild/win32-x64": {
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/win32-x64/-/win32-x64-0.28.2.tgz",
|
||||
"integrity": "sha512-5ebpxr3nWMzrL/rnUI755Jkuee0bHL/Gq0WTF9lvcpv73wAp5eu8MfBUgWK9bhWvZjj7yX8etf/8tI8Ney695g==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"win32"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
}
|
||||
},
|
||||
"node_modules/@modelcontextprotocol/client": {
|
||||
"version": "2.4.0",
|
||||
"resolved": "https://registry.npmjs.org/@modelcontextprotocol/client/-/client-2.4.0.tgz",
|
||||
"integrity": "sha512-LgrD2OphHuG12vfKFKz6SOnq7bKHQmhGJIAYdD+MZBn2npHEyry6uD8zvktodtl5CQM85MO0anUyD9JwmUmxww==",
|
||||
"license": "Apache-2.0",
|
||||
"dependencies": {
|
||||
"@modelcontextprotocol/core": "2.4.0",
|
||||
"cross-spawn": "^7.0.5",
|
||||
"eventsource": "^3.0.2",
|
||||
"eventsource-parser": "^3.0.8",
|
||||
"jose": "^6.1.3",
|
||||
"pkce-challenge": "^5.0.0",
|
||||
"zod": "^4.2.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=20"
|
||||
}
|
||||
},
|
||||
"node_modules/@modelcontextprotocol/core": {
|
||||
"version": "2.4.0",
|
||||
"resolved": "https://registry.npmjs.org/@modelcontextprotocol/core/-/core-2.4.0.tgz",
|
||||
"integrity": "sha512-z8QBKqdfXLPzkLQuG4xp8hrVEMm1PZ1IgxMN2sMbJiFiAKOWXnAzogXn2W4aqIBlP1ea6U5yiqVquvUNM1Cj6Q==",
|
||||
"license": "Apache-2.0",
|
||||
"dependencies": {
|
||||
"zod": "^4.2.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=20"
|
||||
}
|
||||
},
|
||||
"node_modules/@modelcontextprotocol/ext-apps": {
|
||||
"version": "2.0.4",
|
||||
"resolved": "https://registry.npmjs.org/@modelcontextprotocol/ext-apps/-/ext-apps-2.0.4.tgz",
|
||||
"integrity": "sha512-B+PcbuGiaf0e5I3uy4iO5tS3Abo+5w8JvZFzHMzQdq9yVwP836zJkw2+KCGOQ0r4TM1fuQi4JdnbuWjkS4aBvg==",
|
||||
"license": "MIT",
|
||||
"workspaces": [
|
||||
"examples/*"
|
||||
],
|
||||
"dependencies": {
|
||||
"@standard-schema/spec": "^1.1.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=20"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"@modelcontextprotocol/client": "^2.0.0",
|
||||
"@modelcontextprotocol/core": "^2.0.0",
|
||||
"@modelcontextprotocol/server": "^2.0.0",
|
||||
"react": "^17.0.0 || ^18.0.0 || ^19.0.0",
|
||||
"react-dom": "^17.0.0 || ^18.0.0 || ^19.0.0",
|
||||
"zod": "^4.2.0"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"@modelcontextprotocol/server": {
|
||||
"optional": true
|
||||
},
|
||||
"react": {
|
||||
"optional": true
|
||||
},
|
||||
"react-dom": {
|
||||
"optional": true
|
||||
}
|
||||
}
|
||||
},
|
||||
"node_modules/@modelcontextprotocol/server": {
|
||||
"version": "2.4.0",
|
||||
"resolved": "https://registry.npmjs.org/@modelcontextprotocol/server/-/server-2.4.0.tgz",
|
||||
"integrity": "sha512-rgR9yxSaAJ+bcsi/XroT/S2AfPiIWloXAnrdfYSCms9ivVju0ishu/wW0J21Ho6+ZOR1r7RyeRbjiRuF87ix5Q==",
|
||||
"license": "Apache-2.0",
|
||||
"dependencies": {
|
||||
"@modelcontextprotocol/core": "2.4.0",
|
||||
"zod": "^4.2.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=20"
|
||||
}
|
||||
},
|
||||
"node_modules/@napi-rs/keyring": {
|
||||
"version": "2.1.2",
|
||||
"resolved": "https://registry.npmjs.org/@napi-rs/keyring/-/keyring-2.1.2.tgz",
|
||||
"integrity": "sha512-9i1SR6tYeQ3o1beLeVMxhN6Fq513m9TZlTs+46+4JgvSNkaFoW9S4WrpV5MjzE+0CzVoC+dZ4RfZDIgW6+afnQ==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">= 10"
|
||||
},
|
||||
"funding": {
|
||||
"type": "github",
|
||||
"url": "https://github.com/sponsors/Brooooooklyn"
|
||||
},
|
||||
"optionalDependencies": {
|
||||
"@napi-rs/keyring-darwin-arm64": "2.1.2",
|
||||
"@napi-rs/keyring-darwin-x64": "2.1.2",
|
||||
"@napi-rs/keyring-freebsd-x64": "2.1.2",
|
||||
"@napi-rs/keyring-linux-arm-gnueabihf": "2.1.2",
|
||||
"@napi-rs/keyring-linux-arm64-gnu": "2.1.2",
|
||||
"@napi-rs/keyring-linux-arm64-musl": "2.1.2",
|
||||
"@napi-rs/keyring-linux-riscv64-gnu": "2.1.2",
|
||||
"@napi-rs/keyring-linux-x64-gnu": "2.1.2",
|
||||
"@napi-rs/keyring-linux-x64-musl": "2.1.2",
|
||||
"@napi-rs/keyring-win32-arm64-msvc": "2.1.2",
|
||||
"@napi-rs/keyring-win32-ia32-msvc": "2.1.2",
|
||||
"@napi-rs/keyring-win32-x64-msvc": "2.1.2"
|
||||
}
|
||||
},
|
||||
"node_modules/@napi-rs/keyring-darwin-arm64": {
|
||||
"version": "2.1.2",
|
||||
"resolved": "https://registry.npmjs.org/@napi-rs/keyring-darwin-arm64/-/keyring-darwin-arm64-2.1.2.tgz",
|
||||
"integrity": "sha512-vsInUWUzWS6/WzR/TFNwmdtmwKWjnnR8TOpvZ/pjjxDAkJB/waz37PWzp2b8AFcw1aCcq2aQ7k2vvbLQy8pTrw==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"darwin"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">= 10"
|
||||
}
|
||||
},
|
||||
"node_modules/@napi-rs/keyring-darwin-x64": {
|
||||
"version": "2.1.2",
|
||||
"resolved": "https://registry.npmjs.org/@napi-rs/keyring-darwin-x64/-/keyring-darwin-x64-2.1.2.tgz",
|
||||
"integrity": "sha512-wQihYroaQ3r++R6u5c9Gj0TCw1l1F53LCb5aLOXfVvT4DsnDnpe7p7VGYo4/33bLe0iqVevtAj+ET3yKCtOlJQ==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"darwin"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">= 10"
|
||||
}
|
||||
},
|
||||
"node_modules/@napi-rs/keyring-freebsd-x64": {
|
||||
"version": "2.1.2",
|
||||
"resolved": "https://registry.npmjs.org/@napi-rs/keyring-freebsd-x64/-/keyring-freebsd-x64-2.1.2.tgz",
|
||||
"integrity": "sha512-Ve5ha7MbJsq+4TYyXYiTy4IeZOnYxkBB05uH2+nIrEKBkHmyfHttF814EfWAfoJa63KqBhYRXdYJo0o1WVL2sg==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"freebsd"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">= 10"
|
||||
}
|
||||
},
|
||||
"node_modules/@napi-rs/keyring-linux-arm-gnueabihf": {
|
||||
"version": "2.1.2",
|
||||
"resolved": "https://registry.npmjs.org/@napi-rs/keyring-linux-arm-gnueabihf/-/keyring-linux-arm-gnueabihf-2.1.2.tgz",
|
||||
"integrity": "sha512-zxijTnVnTApEyJMjgnLdVQL1g1DAudku/5brrWNDRWj4J0q0nFhUZEtzUA+nY2FvgDjl4s+2YCZYXSTH4TFMUg==",
|
||||
"cpu": [
|
||||
"arm"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">= 10"
|
||||
}
|
||||
},
|
||||
"node_modules/@napi-rs/keyring-linux-arm64-gnu": {
|
||||
"version": "2.1.2",
|
||||
"resolved": "https://registry.npmjs.org/@napi-rs/keyring-linux-arm64-gnu/-/keyring-linux-arm64-gnu-2.1.2.tgz",
|
||||
"integrity": "sha512-PM+SAcl/yGzgGbeeCsKrnlG4T5DBIDjmDqtKm5an1bJz0o0gwSzR3BMiXf04JenkcqascZpdcIReoEvLj0cvCw==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"libc": [
|
||||
"glibc"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">= 10"
|
||||
}
|
||||
},
|
||||
"node_modules/@napi-rs/keyring-linux-arm64-musl": {
|
||||
"version": "2.1.2",
|
||||
"resolved": "https://registry.npmjs.org/@napi-rs/keyring-linux-arm64-musl/-/keyring-linux-arm64-musl-2.1.2.tgz",
|
||||
"integrity": "sha512-+ERS9oyInPCALN957cViBDbYJc724/Q+rZ8ZAz4rOHj4B4v0VR4OAWThdYZgZqD54sZVV0MdyX24qva97hYoAQ==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"libc": [
|
||||
"musl"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">= 10"
|
||||
}
|
||||
},
|
||||
"node_modules/@napi-rs/keyring-linux-riscv64-gnu": {
|
||||
"version": "2.1.2",
|
||||
"resolved": "https://registry.npmjs.org/@napi-rs/keyring-linux-riscv64-gnu/-/keyring-linux-riscv64-gnu-2.1.2.tgz",
|
||||
"integrity": "sha512-pGADtqeT3U1SaFZGjkVsmZVjeNOQx0fr77S+NWszXjZboyaUOLlGKgqG6ssAWzcwxv8uPhKc9Ij17ujNZhSqLw==",
|
||||
"cpu": [
|
||||
"riscv64"
|
||||
],
|
||||
"libc": [
|
||||
"glibc"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">= 10"
|
||||
}
|
||||
},
|
||||
"node_modules/@napi-rs/keyring-linux-x64-gnu": {
|
||||
"version": "2.1.2",
|
||||
"resolved": "https://registry.npmjs.org/@napi-rs/keyring-linux-x64-gnu/-/keyring-linux-x64-gnu-2.1.2.tgz",
|
||||
"integrity": "sha512-Fj5TmTnl2oriuCT5DDEYlhgxYes20uGfgOg10LK9PyPVsI8/PQ1m8I4rq/4A5hkrMtY7wvFb70YmX/yPHDYWvQ==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"libc": [
|
||||
"glibc"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">= 10"
|
||||
}
|
||||
},
|
||||
"node_modules/@napi-rs/keyring-linux-x64-musl": {
|
||||
"version": "2.1.2",
|
||||
"resolved": "https://registry.npmjs.org/@napi-rs/keyring-linux-x64-musl/-/keyring-linux-x64-musl-2.1.2.tgz",
|
||||
"integrity": "sha512-C+KhAeU6x2A43FV3dOmMvUraJb8zsC9VZfsunjZ1cgtuAb208TALaUuAY0jThOphXqhyhBJOs2xTCQhOrRkJIw==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"libc": [
|
||||
"musl"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">= 10"
|
||||
}
|
||||
},
|
||||
"node_modules/@napi-rs/keyring-win32-arm64-msvc": {
|
||||
"version": "2.1.2",
|
||||
"resolved": "https://registry.npmjs.org/@napi-rs/keyring-win32-arm64-msvc/-/keyring-win32-arm64-msvc-2.1.2.tgz",
|
||||
"integrity": "sha512-kBq80ZZfjXoa5wHG9kgAVhYYjaPOLgHh5ngt7r1ftwFXPXDPXKH7oM2r2j2Q64cs0pVtmodVtn5Q9e6xCwzQUQ==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"win32"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">= 10"
|
||||
}
|
||||
},
|
||||
"node_modules/@napi-rs/keyring-win32-ia32-msvc": {
|
||||
"version": "2.1.2",
|
||||
"resolved": "https://registry.npmjs.org/@napi-rs/keyring-win32-ia32-msvc/-/keyring-win32-ia32-msvc-2.1.2.tgz",
|
||||
"integrity": "sha512-bhgKvVbWlZ6iR/tmvb2ht6/o0OzarUX9W7hjg+G+eVdGjR1prwtQ9lYo5ZESdn/+zyF1MIFvHkOCGja9YqDcww==",
|
||||
"cpu": [
|
||||
"ia32"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"win32"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">= 10"
|
||||
}
|
||||
},
|
||||
"node_modules/@napi-rs/keyring-win32-x64-msvc": {
|
||||
"version": "2.1.2",
|
||||
"resolved": "https://registry.npmjs.org/@napi-rs/keyring-win32-x64-msvc/-/keyring-win32-x64-msvc-2.1.2.tgz",
|
||||
"integrity": "sha512-yQtFdfuwqM0ezKlYO6r2n44in2LGzd4MC679Ddhvojip3t1hn+q/4I+gJU+cIJQVd5FAZSKNrK4tGfew3UTpAQ==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"win32"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">= 10"
|
||||
}
|
||||
},
|
||||
"node_modules/@standard-schema/spec": {
|
||||
"version": "1.1.0",
|
||||
"resolved": "https://registry.npmjs.org/@standard-schema/spec/-/spec-1.1.0.tgz",
|
||||
"integrity": "sha512-l2aFy5jALhniG5HgqrD6jXLi/rUWrKvqN/qJx6yoJsgKhblVd+iqqU4RCXavm/jPityDo5TCvKMnpjKnOriy0w==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/@types/node": {
|
||||
"version": "22.19.15",
|
||||
"resolved": "https://registry.npmjs.org/@types/node/-/node-22.19.15.tgz",
|
||||
"integrity": "sha512-F0R/h2+dsy5wJAUe3tAU6oqa2qbWY5TpNfL/RGmo1y38hiyO1w3x2jPtt76wmuaJI4DQnOBu21cNXQ2STIUUWg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"undici-types": "~6.21.0"
|
||||
}
|
||||
},
|
||||
"node_modules/cross-spawn": {
|
||||
"version": "7.0.6",
|
||||
"resolved": "https://registry.npmjs.org/cross-spawn/-/cross-spawn-7.0.6.tgz",
|
||||
"integrity": "sha512-uV2QOWP2nWzsy2aMp8aRibhi9dlzF5Hgh5SHaB9OiTGEyDTiJJyx0uy51QXdyWbtAHNua4XJzUKca3OzKUd3vA==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"path-key": "^3.1.0",
|
||||
"shebang-command": "^2.0.0",
|
||||
"which": "^2.0.1"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 8"
|
||||
}
|
||||
},
|
||||
"node_modules/esbuild": {
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/esbuild/-/esbuild-0.28.2.tgz",
|
||||
"integrity": "sha512-HKVLS8dvII+xoKW9kmqxbRKrnWEXfJJr/FZhhJmiqIB0e053QNYFqOBouTMO/k5sID4MvCiUCvv8b9M4h32wIA==",
|
||||
"dev": true,
|
||||
"hasInstallScript": true,
|
||||
"license": "MIT",
|
||||
"bin": {
|
||||
"esbuild": "bin/esbuild"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
},
|
||||
"optionalDependencies": {
|
||||
"@esbuild/aix-ppc64": "0.28.2",
|
||||
"@esbuild/android-arm": "0.28.2",
|
||||
"@esbuild/android-arm64": "0.28.2",
|
||||
"@esbuild/android-x64": "0.28.2",
|
||||
"@esbuild/darwin-arm64": "0.28.2",
|
||||
"@esbuild/darwin-x64": "0.28.2",
|
||||
"@esbuild/freebsd-arm64": "0.28.2",
|
||||
"@esbuild/freebsd-x64": "0.28.2",
|
||||
"@esbuild/linux-arm": "0.28.2",
|
||||
"@esbuild/linux-arm64": "0.28.2",
|
||||
"@esbuild/linux-ia32": "0.28.2",
|
||||
"@esbuild/linux-loong64": "0.28.2",
|
||||
"@esbuild/linux-mips64el": "0.28.2",
|
||||
"@esbuild/linux-ppc64": "0.28.2",
|
||||
"@esbuild/linux-riscv64": "0.28.2",
|
||||
"@esbuild/linux-s390x": "0.28.2",
|
||||
"@esbuild/linux-x64": "0.28.2",
|
||||
"@esbuild/netbsd-arm64": "0.28.2",
|
||||
"@esbuild/netbsd-x64": "0.28.2",
|
||||
"@esbuild/openbsd-arm64": "0.28.2",
|
||||
"@esbuild/openbsd-x64": "0.28.2",
|
||||
"@esbuild/openharmony-arm64": "0.28.2",
|
||||
"@esbuild/sunos-x64": "0.28.2",
|
||||
"@esbuild/win32-arm64": "0.28.2",
|
||||
"@esbuild/win32-ia32": "0.28.2",
|
||||
"@esbuild/win32-x64": "0.28.2"
|
||||
}
|
||||
},
|
||||
"node_modules/eventsource": {
|
||||
"version": "3.0.7",
|
||||
"resolved": "https://registry.npmjs.org/eventsource/-/eventsource-3.0.7.tgz",
|
||||
"integrity": "sha512-CRT1WTyuQoD771GW56XEZFQ/ZoSfWid1alKGDYMmkt2yl8UXrVR4pspqWNEcqKvVIzg6PAltWjxcSSPrboA4iA==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"eventsource-parser": "^3.0.1"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=18.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/eventsource-parser": {
|
||||
"version": "3.1.1",
|
||||
"resolved": "https://registry.npmjs.org/eventsource-parser/-/eventsource-parser-3.1.1.tgz",
|
||||
"integrity": "sha512-EKN1vKAMcZ8MlYMpaNuxN6R9yakzH6uajHcHVTqWJzvu5pWw9DyhbP35HH8MVBQ+dZjAfDxk+A8NiR9KWaXiyQ==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=18.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/isexe": {
|
||||
"version": "2.0.0",
|
||||
"resolved": "https://registry.npmjs.org/isexe/-/isexe-2.0.0.tgz",
|
||||
"integrity": "sha512-RHxMLp9lnKHGHRng9QFhRCMbYAcVpn69smSGcq3f36xjgVVWThj4qqLbTLlq7Ssj8B+fIQ1EuCEGI2lKsyQeIw==",
|
||||
"license": "ISC"
|
||||
},
|
||||
"node_modules/jose": {
|
||||
"version": "6.2.15",
|
||||
"resolved": "https://registry.npmjs.org/jose/-/jose-6.2.15.tgz",
|
||||
"integrity": "sha512-qdF6ewcZARtef63fJ9AKKZPgxw+t7USH6HLXYcD4CrdmSvDp79rpJSPFEKc90QBeGS63u1KQtN2MZQHC0yDIzA==",
|
||||
"license": "MIT",
|
||||
"funding": {
|
||||
"url": "https://github.com/sponsors/panva"
|
||||
}
|
||||
},
|
||||
"node_modules/path-key": {
|
||||
"version": "3.1.1",
|
||||
"resolved": "https://registry.npmjs.org/path-key/-/path-key-3.1.1.tgz",
|
||||
"integrity": "sha512-ojmeN0qd+y0jszEtoY48r0Peq5dwMEkIlCOu6Q5f41lfkswXuKtYrhgoTpLnyIcHm24Uhqx+5Tqm2InSwLhE6Q==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=8"
|
||||
}
|
||||
},
|
||||
"node_modules/pkce-challenge": {
|
||||
"version": "5.0.1",
|
||||
"resolved": "https://registry.npmjs.org/pkce-challenge/-/pkce-challenge-5.0.1.tgz",
|
||||
"integrity": "sha512-wQ0b/W4Fr01qtpHlqSqspcj3EhBvimsdh0KlHhH8HRZnMsEa0ea2fTULOXOS9ccQr3om+GcGRk4e+isrZWV8qQ==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=16.20.0"
|
||||
}
|
||||
},
|
||||
"node_modules/shebang-command": {
|
||||
"version": "2.0.0",
|
||||
"resolved": "https://registry.npmjs.org/shebang-command/-/shebang-command-2.0.0.tgz",
|
||||
"integrity": "sha512-kHxr2zZpYtdmrN1qDjrrX/Z1rR1kG8Dx+gkpK1G4eXmvXswmcE1hTWBWYUzlraYw1/yZp6YuDY77YtvbN0dmDA==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"shebang-regex": "^3.0.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=8"
|
||||
}
|
||||
},
|
||||
"node_modules/shebang-regex": {
|
||||
"version": "3.0.0",
|
||||
"resolved": "https://registry.npmjs.org/shebang-regex/-/shebang-regex-3.0.0.tgz",
|
||||
"integrity": "sha512-7++dFhtcx3353uBaq8DDR4NuxBetBzC7ZQOhmTQInHEd6bSrXdiEyzCvG07Z44UYdLShWUyXt5M/yhz8ekcb1A==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=8"
|
||||
}
|
||||
},
|
||||
"node_modules/typescript": {
|
||||
"version": "5.9.3",
|
||||
"resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz",
|
||||
"integrity": "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==",
|
||||
"dev": true,
|
||||
"license": "Apache-2.0",
|
||||
"bin": {
|
||||
"tsc": "bin/tsc",
|
||||
"tsserver": "bin/tsserver"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=14.17"
|
||||
}
|
||||
},
|
||||
"node_modules/undici-types": {
|
||||
"version": "6.21.0",
|
||||
"resolved": "https://registry.npmjs.org/undici-types/-/undici-types-6.21.0.tgz",
|
||||
"integrity": "sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/which": {
|
||||
"version": "2.0.2",
|
||||
"resolved": "https://registry.npmjs.org/which/-/which-2.0.2.tgz",
|
||||
"integrity": "sha512-BLI3Tl1TW3Pvl70l3yq3Y64i+awpwXqsGBYWkkqMtnbXgrMD+yj7rhW0kuEDxzJaYXGjEW5ogapKNMEKNMjibA==",
|
||||
"license": "ISC",
|
||||
"dependencies": {
|
||||
"isexe": "^2.0.0"
|
||||
},
|
||||
"bin": {
|
||||
"node-which": "bin/node-which"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 8"
|
||||
}
|
||||
},
|
||||
"node_modules/zod": {
|
||||
"version": "4.6.7",
|
||||
"resolved": "https://registry.npmjs.org/zod/-/zod-4.6.7.tgz",
|
||||
"integrity": "sha512-g8W2D2kU6a8untc053fX1uVBKzM7yqAxW/3zwk83+6RyF2+SKOuV+c0wBDhUwzV1PaGapmZH+rmcUzPf+3CGwg==",
|
||||
"license": "MIT",
|
||||
"funding": {
|
||||
"url": "https://github.com/sponsors/colinhacks"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
{
|
||||
"name": "@chunkdash/crossdesk-mcp",
|
||||
"version": "0.1.0",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"description": "Local CrossDesk MCP tools and remote desktop panels",
|
||||
"license": "GPL-3.0-only",
|
||||
"author": "DI JUNKUN",
|
||||
"repository": {
|
||||
"type": "git",
|
||||
"url": "git+https://github.com/kunkundi/crossdesk.git",
|
||||
"directory": "apps/mcp"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=22"
|
||||
},
|
||||
"bin": {
|
||||
"crossdesk-mcp": "dist/main.js"
|
||||
},
|
||||
"scripts": {
|
||||
"build": "tsc && tsc -p tsconfig.ui.json && node scripts/build-ui.mjs",
|
||||
"check": "tsc --noEmit && tsc -p tsconfig.ui.json",
|
||||
"pack:npm": "node scripts/pack.mjs"
|
||||
},
|
||||
"dependencies": {
|
||||
"@modelcontextprotocol/client": "2.4.0",
|
||||
"@modelcontextprotocol/core": "2.4.0",
|
||||
"@modelcontextprotocol/ext-apps": "2.0.4",
|
||||
"@modelcontextprotocol/server": "2.4.0",
|
||||
"@napi-rs/keyring": "2.1.2",
|
||||
"zod": "4.6.7"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "22.19.15",
|
||||
"esbuild": "0.28.2",
|
||||
"typescript": "5.9.3"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,35 @@
|
||||
import { build } from "esbuild";
|
||||
import { mkdir, readFile, writeFile } from "node:fs/promises";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { dirname, resolve } from "node:path";
|
||||
|
||||
const root = resolve(dirname(fileURLToPath(import.meta.url)), "..");
|
||||
// Unmodified apps/desktop/resources/linux/crossdesk_128x128.png.
|
||||
// Keep a local copy so the npm source package rebuilds without desktop assets.
|
||||
const logo = "data:image/png;base64," +
|
||||
(await readFile(resolve(root, "ui/assets/crossdesk.png"))).toString("base64");
|
||||
await mkdir(resolve(root, "dist"), { recursive: true });
|
||||
for (const mode of ["apps", "browser"]) {
|
||||
const result = await build({
|
||||
entryPoints: [resolve(root, "ui/viewer.ts")],
|
||||
bundle: true,
|
||||
minify: true,
|
||||
platform: "browser",
|
||||
target: "es2022",
|
||||
outfile: "viewer.js",
|
||||
write: false,
|
||||
define: {
|
||||
CROSSDESK_APPS: String(mode === "apps"),
|
||||
CROSSDESK_LOGO: JSON.stringify(logo),
|
||||
},
|
||||
});
|
||||
const script = result.outputFiles
|
||||
.find((f) => f.path.endsWith(".js"))
|
||||
.text.replaceAll("</script", "<\\/script");
|
||||
const css = result.outputFiles.find((f) => f.path.endsWith(".css")).text;
|
||||
const nonce = mode === "browser" ? ' nonce="__CROSSDESK_NONCE__"' : "";
|
||||
await writeFile(
|
||||
resolve(root, `dist/viewer-${mode}.html`),
|
||||
`<!doctype html><html lang="zh-CN"><head><meta charset="UTF-8"><meta name="viewport" content="width=device-width,initial-scale=1"><meta name="theme-color" content="#ffffff"><link rel="icon" type="image/png" href="${logo}"><title>CrossDesk MCP</title><style${nonce}>${css}</style></head><body><div id="app"></div><script${nonce}>${script}</script></body></html>`,
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,107 @@
|
||||
import {
|
||||
mkdtemp,
|
||||
readFile,
|
||||
writeFile,
|
||||
mkdir,
|
||||
cp,
|
||||
chmod,
|
||||
rm,
|
||||
access,
|
||||
} from "node:fs/promises";
|
||||
import { dirname, resolve, join } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { tmpdir } from "node:os";
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { parseArgs } from "node:util";
|
||||
|
||||
const root = resolve(dirname(fileURLToPath(import.meta.url)), "..");
|
||||
const repository = resolve(root, "../..");
|
||||
const { values } = parseArgs({ options: { "out-dir": { type: "string" } } });
|
||||
const output = resolve(values["out-dir"] ?? join(repository, "build/npm-mcp"));
|
||||
const project = JSON.parse(await readFile(join(root, "package.json"), "utf8"));
|
||||
const cli = JSON.parse(
|
||||
await readFile(join(root, "../cli/package.json"), "utf8"),
|
||||
);
|
||||
const platforms = JSON.parse(
|
||||
await readFile(join(root, "../cli/npm/platforms.json"), "utf8"),
|
||||
);
|
||||
const native = await import(new URL("../dist/native.js", import.meta.url));
|
||||
if (native.NATIVE_VERSION !== cli.version)
|
||||
throw new Error("MCP and CLI native versions must match.");
|
||||
for (const file of ["main.js", "viewer-browser.html", "viewer-apps.html"])
|
||||
await access(join(root, "dist", file));
|
||||
const staging = await mkdtemp(join(tmpdir(), "crossdesk-mcp-pack-"));
|
||||
try {
|
||||
for (const entry of [
|
||||
"dist",
|
||||
"src",
|
||||
"ui",
|
||||
"scripts",
|
||||
"tsconfig.json",
|
||||
"tsconfig.ui.json",
|
||||
"package-lock.json",
|
||||
"README.md",
|
||||
])
|
||||
await cp(join(root, entry), join(staging, entry), { recursive: true });
|
||||
await cp(join(repository, "LICENSE"), join(staging, "LICENSE"));
|
||||
await writeFile(
|
||||
join(staging, "build-package.json"),
|
||||
JSON.stringify(project, null, 2) + "\n",
|
||||
);
|
||||
const manifest = {
|
||||
...project,
|
||||
private: false,
|
||||
files: [
|
||||
"dist",
|
||||
"src",
|
||||
"ui",
|
||||
"scripts/build-ui.mjs",
|
||||
"tsconfig.json",
|
||||
"tsconfig.ui.json",
|
||||
"build-package.json",
|
||||
"package-lock.json",
|
||||
"README.md",
|
||||
"LICENSE",
|
||||
],
|
||||
optionalDependencies: Object.fromEntries(
|
||||
Object.values(platforms).map((p) => [p.packageName, cli.version]),
|
||||
),
|
||||
publishConfig: { access: "public" },
|
||||
};
|
||||
delete manifest.scripts;
|
||||
delete manifest.devDependencies;
|
||||
await writeFile(
|
||||
join(staging, "package.json"),
|
||||
JSON.stringify(manifest, null, 2) + "\n",
|
||||
);
|
||||
await chmod(join(staging, "dist/main.js"), 0o755);
|
||||
await mkdir(output, { recursive: true });
|
||||
const filename = `${project.name.replace(/^@/, "").replace("/", "-")}-${project.version}.tgz`;
|
||||
try {
|
||||
await access(join(output, filename));
|
||||
throw new Error(`Output already exists: ${filename}`);
|
||||
} catch (error) {
|
||||
if (error.code !== "ENOENT") throw error;
|
||||
}
|
||||
const args = [
|
||||
"pack",
|
||||
"--json",
|
||||
"--ignore-scripts",
|
||||
"--pack-destination",
|
||||
output,
|
||||
];
|
||||
if (process.platform === "win32" && !process.env.npm_execpath)
|
||||
throw new Error("On Windows use npm run pack:npm.");
|
||||
const result = execFileSync(
|
||||
process.env.npm_execpath ? process.execPath : "npm",
|
||||
process.env.npm_execpath ? [process.env.npm_execpath, ...args] : args,
|
||||
{ cwd: staging, encoding: "utf8" },
|
||||
);
|
||||
const parsed = JSON.parse(result);
|
||||
const packages = Array.isArray(parsed) ? parsed : Object.values(parsed);
|
||||
if (packages.length !== 1 || packages[0].filename !== filename)
|
||||
throw new Error("Unexpected package archive.");
|
||||
process.stdout.write(join(output, filename) + "\n");
|
||||
} finally {
|
||||
await rm(staging, { recursive: true, force: true });
|
||||
}
|
||||
@@ -0,0 +1,406 @@
|
||||
import {
|
||||
createServer,
|
||||
type IncomingMessage,
|
||||
type ServerResponse,
|
||||
type Server,
|
||||
} from "node:http";
|
||||
import {
|
||||
createServer as createIPC,
|
||||
createConnection,
|
||||
type Server as IPCServer,
|
||||
} from "node:net";
|
||||
import { spawn } from "node:child_process";
|
||||
import { readFile, writeFile, unlink, chmod, readdir } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { Failure, token, equal, failure, type Data } from "./common.js";
|
||||
import { Sessions, type View } from "./sessions.js";
|
||||
import { uiAction } from "./ui-actions.js";
|
||||
|
||||
const uiFile = fileURLToPath(new URL("./viewer-browser.html", import.meta.url));
|
||||
interface BrowserSession {
|
||||
csrf: string;
|
||||
views: Set<string>;
|
||||
expires: number;
|
||||
}
|
||||
export async function openURL(url: string): Promise<boolean> {
|
||||
if (
|
||||
process.platform === "linux" &&
|
||||
!process.env.DISPLAY &&
|
||||
!process.env.WAYLAND_DISPLAY
|
||||
)
|
||||
return false;
|
||||
const command =
|
||||
process.platform === "darwin"
|
||||
? "open"
|
||||
: process.platform === "win32"
|
||||
? "rundll32.exe"
|
||||
: "xdg-open";
|
||||
const args =
|
||||
process.platform === "win32" ? ["url.dll,FileProtocolHandler", url] : [url];
|
||||
return new Promise((resolve) => {
|
||||
const child = spawn(command, args, {
|
||||
stdio: "ignore",
|
||||
shell: false,
|
||||
windowsHide: true,
|
||||
});
|
||||
const timeout = setTimeout(() => {
|
||||
child.unref();
|
||||
resolve(true);
|
||||
}, 1500);
|
||||
child.once("error", () => {
|
||||
clearTimeout(timeout);
|
||||
resolve(false);
|
||||
});
|
||||
child.once("exit", (code) => {
|
||||
clearTimeout(timeout);
|
||||
resolve(code === 0);
|
||||
});
|
||||
});
|
||||
}
|
||||
export class BrowserPanel {
|
||||
private closed = false;
|
||||
private server?: Server;
|
||||
private starting?: Promise<void>;
|
||||
private ipc?: IPCServer;
|
||||
private ipcPath?: string;
|
||||
private descriptor?: string;
|
||||
private origin = "";
|
||||
private cookie = "crossdesk_" + token();
|
||||
private grants = new Map<string, { view: string; expires: number }>();
|
||||
private clients = new Map<string, BrowserSession>();
|
||||
private html = "";
|
||||
constructor(private sessions: Sessions) {}
|
||||
async connectDomains() {
|
||||
await this.start();
|
||||
// Native processes own ephemeral video ports, including after reconnect.
|
||||
// Restrict the wildcard to ports on the literal loopback address.
|
||||
return [this.origin, "ws://127.0.0.1:*"];
|
||||
}
|
||||
async video(view: View) {
|
||||
await this.start();
|
||||
if (this.closed) throw new Failure("PANEL_CLOSED");
|
||||
return this.sessions.openVideo(view, this.origin);
|
||||
}
|
||||
async open(view: View): Promise<Data> {
|
||||
if (this.closed) throw new Failure("PANEL_CLOSED");
|
||||
await this.start();
|
||||
if (this.closed) throw new Failure("PANEL_CLOSED");
|
||||
const opened = await openURL(this.bootstrapURL(view));
|
||||
return {
|
||||
...this.sessions.describe(view),
|
||||
browser_opened: opened,
|
||||
browser_url: this.origin,
|
||||
pairing_command: {
|
||||
command: "crossdesk-mcp",
|
||||
args: [
|
||||
"setup",
|
||||
"--open-panel",
|
||||
"--view",
|
||||
view.id,
|
||||
"--config-dir",
|
||||
this.sessions.store.directory,
|
||||
],
|
||||
},
|
||||
};
|
||||
}
|
||||
private bootstrapURL(view: View): string {
|
||||
if (this.closed) throw new Failure("PANEL_CLOSED");
|
||||
for (const [key, value] of this.grants)
|
||||
if (value.expires < Date.now()) this.grants.delete(key);
|
||||
if (this.grants.size >= 32) throw new Failure("VIEW_LIMIT");
|
||||
const grant = token();
|
||||
this.grants.set(grant, { view: view.id, expires: Date.now() + 60000 });
|
||||
return `${this.origin}/#bootstrap=${grant}`;
|
||||
}
|
||||
private async start() {
|
||||
this.starting ??= (async () => {
|
||||
this.html = await readFile(uiFile, "utf8");
|
||||
if (this.closed) throw new Failure("PANEL_CLOSED");
|
||||
this.server = createServer((req, res) => {
|
||||
void this.handle(req, res).catch(() => {
|
||||
if (!res.headersSent)
|
||||
this.respond(res, 500, { error: { code: "INTERNAL_ERROR" } });
|
||||
else res.end();
|
||||
});
|
||||
});
|
||||
// No upgrade handler or video bytes in the Node HTTP server.
|
||||
this.server.requestTimeout = 45000;
|
||||
this.server.headersTimeout = 10000;
|
||||
this.server.maxHeadersCount = 32;
|
||||
await new Promise<void>((resolve, reject) => {
|
||||
this.server!.once("error", reject);
|
||||
this.server!.listen(0, "127.0.0.1", resolve);
|
||||
});
|
||||
this.origin = `http://127.0.0.1:${(this.server.address() as import("node:net").AddressInfo).port}`;
|
||||
if (this.closed) throw new Failure("PANEL_CLOSED");
|
||||
await this.startIPC();
|
||||
if (this.closed) throw new Failure("PANEL_CLOSED");
|
||||
})().catch(async (error) => {
|
||||
await this.close(false);
|
||||
this.starting = undefined;
|
||||
throw error;
|
||||
});
|
||||
return this.starting;
|
||||
}
|
||||
private respond(res: ServerResponse, code: number, value: Data) {
|
||||
res.writeHead(code, { "Content-Type": "application/json; charset=utf-8" });
|
||||
res.end(JSON.stringify(value));
|
||||
}
|
||||
private client(req: IncomingMessage): BrowserSession | undefined {
|
||||
const value = req.headers.cookie
|
||||
?.split(";")
|
||||
.map((s) => s.trim())
|
||||
.find((s) => s.startsWith(this.cookie + "="))
|
||||
?.slice(this.cookie.length + 1);
|
||||
const client = value ? this.clients.get(value) : undefined;
|
||||
return client && client.expires > Date.now() ? client : undefined;
|
||||
}
|
||||
private async body(req: IncomingMessage): Promise<Data> {
|
||||
if (req.headers["content-type"] !== "application/json")
|
||||
throw new Failure("CONTENT_TYPE_REQUIRED");
|
||||
let bytes = 0;
|
||||
const chunks: Buffer[] = [];
|
||||
for await (const chunk of req) {
|
||||
bytes += chunk.length;
|
||||
if (bytes > 65536) throw new Failure("MESSAGE_TOO_LARGE");
|
||||
chunks.push(chunk);
|
||||
}
|
||||
try {
|
||||
const value = JSON.parse(Buffer.concat(chunks).toString("utf8"));
|
||||
if (!value || Array.isArray(value) || typeof value !== "object")
|
||||
throw new Error();
|
||||
return value;
|
||||
} catch {
|
||||
throw new Failure("INVALID_ARGUMENT");
|
||||
}
|
||||
}
|
||||
private async handle(req: IncomingMessage, res: ServerResponse) {
|
||||
res.setHeader("Cache-Control", "no-store");
|
||||
res.setHeader("Referrer-Policy", "no-referrer");
|
||||
res.setHeader("X-Content-Type-Options", "nosniff");
|
||||
res.setHeader("X-Frame-Options", "DENY");
|
||||
const authority = this.origin.slice("http://".length);
|
||||
if (
|
||||
req.headers.host !== authority ||
|
||||
(req.headers.origin && req.headers.origin !== this.origin)
|
||||
)
|
||||
return this.respond(res, 403, { error: { code: "ORIGIN_DENIED" } });
|
||||
if (req.url === "/" && req.method === "GET") {
|
||||
const nonce = token();
|
||||
res.setHeader(
|
||||
"Content-Security-Policy",
|
||||
`default-src 'none'; script-src 'nonce-${nonce}'; style-src 'nonce-${nonce}'; img-src data:; connect-src 'self' ws://127.0.0.1:*; base-uri 'none'; form-action 'none'; frame-ancestors 'none'`,
|
||||
);
|
||||
res.writeHead(200, { "Content-Type": "text/html; charset=utf-8" });
|
||||
res.end(this.html.replaceAll("__CROSSDESK_NONCE__", nonce));
|
||||
return;
|
||||
}
|
||||
if (req.url === "/api/context" && req.method === "GET") {
|
||||
const client = this.client(req);
|
||||
if (!client)
|
||||
return this.respond(res, 401, { error: { code: "PAIRING_REQUIRED" } });
|
||||
for (const id of client.views)
|
||||
if (!this.sessions.views.has(id)) client.views.delete(id);
|
||||
if (!client.views.size)
|
||||
return this.respond(res, 401, { error: { code: "PAIRING_REQUIRED" } });
|
||||
return this.respond(res, 200, {
|
||||
csrf: client.csrf,
|
||||
views: [...client.views].map((v) =>
|
||||
this.sessions.describe(this.sessions.view(v)),
|
||||
),
|
||||
});
|
||||
}
|
||||
if (req.method !== "POST" || req.headers.origin !== this.origin)
|
||||
return this.respond(res, 403, { error: { code: "ORIGIN_DENIED" } });
|
||||
try {
|
||||
const body = await this.body(req);
|
||||
if (req.url === "/api/bootstrap") {
|
||||
const grant =
|
||||
typeof body.token === "string"
|
||||
? this.grants.get(body.token)
|
||||
: undefined;
|
||||
if (!grant || grant.expires < Date.now())
|
||||
throw new Failure("PAIRING_REQUIRED");
|
||||
this.grants.delete(body.token);
|
||||
for (const [id, client] of this.clients)
|
||||
if (client.expires < Date.now()) this.clients.delete(id);
|
||||
const existing = this.client(req);
|
||||
if (!existing && this.clients.size >= 32)
|
||||
throw new Failure("VIEW_LIMIT");
|
||||
const session = token(),
|
||||
client = existing ?? {
|
||||
csrf: token(),
|
||||
views: new Set<string>(),
|
||||
expires: 0,
|
||||
};
|
||||
client.views.add(grant.view);
|
||||
client.expires = Date.now() + 12 * 60 * 60 * 1000;
|
||||
for (const [key, previous] of this.clients)
|
||||
if (previous === client) this.clients.delete(key);
|
||||
this.clients.set(session, client);
|
||||
res.setHeader(
|
||||
"Set-Cookie",
|
||||
`${this.cookie}=${session}; HttpOnly; SameSite=Strict; Path=/api`,
|
||||
);
|
||||
return this.respond(res, 200, {
|
||||
csrf: client.csrf,
|
||||
...this.sessions.describe(this.sessions.view(grant.view)),
|
||||
});
|
||||
}
|
||||
const client = this.client(req);
|
||||
if (
|
||||
!client ||
|
||||
typeof req.headers["x-crossdesk-csrf"] !== "string" ||
|
||||
!equal(client.csrf, req.headers["x-crossdesk-csrf"])
|
||||
)
|
||||
throw new Failure("PANEL_UNAUTHORIZED");
|
||||
if (typeof body.view_id !== "string" || !client.views.has(body.view_id))
|
||||
throw new Failure("VIEW_UNAUTHORIZED");
|
||||
const name = req.url?.match(
|
||||
/^\/api\/(ready|devices|status|connect|video_open|control)$/,
|
||||
)?.[1];
|
||||
if (!name) throw new Failure("UNKNOWN_ACTION");
|
||||
const { view_id, ...args } = body;
|
||||
const controller = new AbortController();
|
||||
res.once("close", () => {
|
||||
if (!res.writableEnded) controller.abort();
|
||||
});
|
||||
const result = name === "video_open" ? await this.video(this.sessions.view(view_id)) : await uiAction(
|
||||
this.sessions,
|
||||
this.sessions.view(view_id),
|
||||
name,
|
||||
args,
|
||||
controller.signal,
|
||||
);
|
||||
this.respond(res, 200, result);
|
||||
} catch (error) {
|
||||
this.respond(res, 400, { error: failure(error) });
|
||||
}
|
||||
}
|
||||
private async startIPC() {
|
||||
const secret = token();
|
||||
const suffix = token().slice(0, 20);
|
||||
this.ipcPath =
|
||||
process.platform === "win32"
|
||||
? `\\\\.\\pipe\\crossdesk-mcp-${suffix}`
|
||||
: join(tmpdir(), `cd-mcp-${suffix}.sock`);
|
||||
this.ipc = createIPC((socket) => {
|
||||
socket.setTimeout(5000, () => socket.destroy());
|
||||
let data = "",
|
||||
consumed = false;
|
||||
socket.on("error", () => {});
|
||||
socket.on("data", (part) => {
|
||||
data += part.toString("utf8");
|
||||
if (data.length > 2048) {
|
||||
socket.destroy();
|
||||
return;
|
||||
}
|
||||
if (consumed || !data.includes("\n")) return;
|
||||
consumed = true;
|
||||
void (async () => {
|
||||
try {
|
||||
const p = JSON.parse(data);
|
||||
if (
|
||||
typeof p.secret !== "string" ||
|
||||
!equal(p.secret, secret) ||
|
||||
typeof p.view_id !== "string"
|
||||
)
|
||||
throw new Failure("PAIRING_DENIED");
|
||||
const v = this.sessions.view(p.view_id);
|
||||
if (
|
||||
v.panel !== "browser" ||
|
||||
!["auto", "browser"].includes(this.sessions.policy)
|
||||
)
|
||||
throw new Failure("PANEL_POLICY_DENIED");
|
||||
// Only the authenticated local setup process receives this URL.
|
||||
// It uses its own desktop environment to open the browser.
|
||||
socket.end(
|
||||
JSON.stringify({ ok: true, url: this.bootstrapURL(v) }) + "\n",
|
||||
);
|
||||
} catch {
|
||||
socket.end('{"ok":false}\n');
|
||||
}
|
||||
})();
|
||||
});
|
||||
});
|
||||
await new Promise<void>((resolve, reject) => {
|
||||
this.ipc!.once("error", reject);
|
||||
this.ipc!.listen(this.ipcPath!, resolve);
|
||||
});
|
||||
if (process.platform !== "win32") await chmod(this.ipcPath, 0o600);
|
||||
this.descriptor = join(
|
||||
this.sessions.store.directory,
|
||||
`run-${process.pid}-${suffix}.json`,
|
||||
);
|
||||
await writeFile(
|
||||
this.descriptor,
|
||||
JSON.stringify({ pid: process.pid, endpoint: this.ipcPath, secret }),
|
||||
{ mode: 0o600, flag: "wx" },
|
||||
);
|
||||
}
|
||||
async close(final = true) {
|
||||
if (final) this.closed = true;
|
||||
await this.sessions.revokeVideo();
|
||||
this.grants.clear();
|
||||
this.clients.clear();
|
||||
this.server?.closeAllConnections();
|
||||
if (this.server?.listening) this.server.close();
|
||||
if (this.ipc?.listening) this.ipc.close();
|
||||
if (this.descriptor) await unlink(this.descriptor).catch(() => {});
|
||||
if (this.ipcPath && process.platform !== "win32")
|
||||
await unlink(this.ipcPath).catch(() => {});
|
||||
}
|
||||
}
|
||||
export async function pairBrowser(
|
||||
directory: string,
|
||||
view: string,
|
||||
): Promise<string | undefined> {
|
||||
for (const file of await readdir(directory)) {
|
||||
if (!/^run-\d+-[a-zA-Z0-9_-]+\.json$/.test(file)) continue;
|
||||
try {
|
||||
const descriptor = JSON.parse(
|
||||
await readFile(join(directory, file), "utf8"),
|
||||
);
|
||||
const url = await new Promise<string | undefined>((resolve) => {
|
||||
const socket = createConnection(descriptor.endpoint);
|
||||
let data = "";
|
||||
socket.setTimeout(5000, () => {
|
||||
socket.destroy();
|
||||
resolve(undefined);
|
||||
});
|
||||
socket.once("error", () => resolve(undefined));
|
||||
socket.once("connect", () =>
|
||||
socket.write(
|
||||
JSON.stringify({ secret: descriptor.secret, view_id: view }) + "\n",
|
||||
),
|
||||
);
|
||||
socket.on("data", (part) => {
|
||||
data += part;
|
||||
if (data.length > 2048) socket.destroy();
|
||||
});
|
||||
socket.once("end", () => {
|
||||
try {
|
||||
const result = JSON.parse(data);
|
||||
resolve(
|
||||
result.ok &&
|
||||
typeof result.url === "string" &&
|
||||
/^http:\/\/127\.0\.0\.1:\d+\/#bootstrap=[a-zA-Z0-9_-]+$/.test(
|
||||
result.url,
|
||||
)
|
||||
? result.url
|
||||
: undefined,
|
||||
);
|
||||
} catch {
|
||||
resolve(undefined);
|
||||
}
|
||||
});
|
||||
});
|
||||
if (url) return url;
|
||||
} catch {
|
||||
/* Another MCP instance may have exited. */
|
||||
}
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
@@ -0,0 +1,131 @@
|
||||
import { randomBytes, timingSafeEqual } from "node:crypto";
|
||||
import { z } from "zod";
|
||||
|
||||
export type Data = Record<string, any>;
|
||||
export const token = () => randomBytes(24).toString("base64url");
|
||||
export const equal = (a: string, b: string) => {
|
||||
const left = Buffer.from(a),
|
||||
right = Buffer.from(b);
|
||||
return left.length === right.length && timingSafeEqual(left, right);
|
||||
};
|
||||
export class Failure extends Error {
|
||||
constructor(
|
||||
public code: string,
|
||||
public sideEffect = false,
|
||||
public retryable = false,
|
||||
) {
|
||||
super(code);
|
||||
}
|
||||
}
|
||||
export function failure(error: unknown): Data {
|
||||
const e = error instanceof Failure ? error : new Failure("INTERNAL_ERROR");
|
||||
return {
|
||||
code: e.code,
|
||||
retryable: e.retryable,
|
||||
side_effect_possible: e.sideEffect,
|
||||
};
|
||||
}
|
||||
export function parse<T extends z.ZodType>(
|
||||
schema: T,
|
||||
value: unknown,
|
||||
): z.output<T> {
|
||||
const result = schema.safeParse(value);
|
||||
if (!result.success) throw new Failure("INVALID_ARGUMENT");
|
||||
return result.data;
|
||||
}
|
||||
export const id = z
|
||||
.string()
|
||||
.min(1)
|
||||
.max(128)
|
||||
.regex(/^[a-zA-Z0-9_-]+$/);
|
||||
export const alias = z
|
||||
.string()
|
||||
.min(1)
|
||||
.max(80)
|
||||
.regex(/^[\p{L}\p{N} _.-]+$/u);
|
||||
export const mode = z.enum(["observe", "control"]);
|
||||
export const panel = z.enum(["auto", "apps", "browser", "off"]);
|
||||
export type Panel = z.infer<typeof panel>;
|
||||
export const videoOptions = z.strictObject({
|
||||
decode_path: z.enum(["browser", "minirtc"]).default("browser"),
|
||||
decoder: z.enum(["auto", "software"]).default("auto"),
|
||||
});
|
||||
export type VideoOptions = z.infer<typeof videoOptions>;
|
||||
export const host = z
|
||||
.string()
|
||||
.min(1)
|
||||
.max(253)
|
||||
.regex(/^[a-zA-Z0-9.:-]+$/);
|
||||
export const point = z.strictObject({
|
||||
x: z.number().finite().min(0).max(8192),
|
||||
y: z.number().finite().min(0).max(8192),
|
||||
});
|
||||
const button = z.enum(["left", "right", "middle"]).optional();
|
||||
const key = z
|
||||
.string()
|
||||
.min(1)
|
||||
.max(16)
|
||||
.regex(/^[A-Z0-9_]+$/);
|
||||
export const action = z.discriminatedUnion("type", [
|
||||
point.extend({ type: z.literal("move") }),
|
||||
point.extend({ type: z.literal("click"), button }),
|
||||
point.extend({ type: z.literal("double_click"), button }),
|
||||
point.extend({
|
||||
type: z.literal("scroll"),
|
||||
direction: z.enum(["up", "down", "left", "right"]),
|
||||
ticks: z.int().min(1).max(10).optional(),
|
||||
}),
|
||||
z.strictObject({
|
||||
type: z.literal("drag"),
|
||||
path: z.array(point).min(2).max(128),
|
||||
duration_ms: z.int().min(100).max(2000).optional(),
|
||||
}),
|
||||
z.strictObject({ type: z.literal("key_press"), key }),
|
||||
z.strictObject({
|
||||
type: z.literal("hotkey"),
|
||||
keys: z.array(key).min(2).max(6),
|
||||
}),
|
||||
z.strictObject({
|
||||
type: z.literal("type_text"),
|
||||
text: z
|
||||
.string()
|
||||
.min(1)
|
||||
.max(16384)
|
||||
.refine((s) => Buffer.byteLength(s) <= 16384 && !s.includes("\0")),
|
||||
}),
|
||||
]);
|
||||
export const screenshot = z.strictObject({
|
||||
session_id: id,
|
||||
region: z
|
||||
.strictObject({
|
||||
x: z.int().min(0),
|
||||
y: z.int().min(0),
|
||||
width: z.int().min(1).max(8192),
|
||||
height: z.int().min(1).max(8192),
|
||||
})
|
||||
.optional(),
|
||||
max_edge: z.int().min(64).max(2048).optional(),
|
||||
after_frame_id: z.int().min(0).optional(),
|
||||
generation: z.int().min(0).optional(),
|
||||
timeout_ms: z.int().min(0).max(5000).optional(),
|
||||
});
|
||||
export const uiConnect = z.strictObject({
|
||||
device_alias: alias.optional(),
|
||||
device_id: z
|
||||
.string()
|
||||
.regex(/^\d{9}$/)
|
||||
.optional(),
|
||||
label: alias.optional(),
|
||||
password: z
|
||||
.string()
|
||||
.max(256)
|
||||
.refine((s) => Buffer.byteLength(s) <= 256 && !/[\0\r\n]/.test(s))
|
||||
.optional(),
|
||||
mode: mode.default("observe"),
|
||||
display_index: z.int().min(0).max(63).default(0),
|
||||
server: host.optional(),
|
||||
port: z.int().min(1).max(65535).optional(),
|
||||
save_device: z.boolean().default(false),
|
||||
remember_password: z.boolean().default(false),
|
||||
persistent_authorization: z.boolean().default(false),
|
||||
});
|
||||
@@ -0,0 +1,196 @@
|
||||
#!/usr/bin/env node
|
||||
import { parseArgs } from "node:util";
|
||||
import { createInterface } from "node:readline/promises";
|
||||
import { Writable } from "node:stream";
|
||||
import {
|
||||
serveStdio,
|
||||
StdioServerTransport,
|
||||
} from "@modelcontextprotocol/server/stdio";
|
||||
import { DeviceStore, type Device } from "./store.js";
|
||||
import { Sessions } from "./sessions.js";
|
||||
import { BrowserPanel, pairBrowser, openURL } from "./browser.js";
|
||||
import { createServer } from "./server.js";
|
||||
import {
|
||||
panel,
|
||||
parse,
|
||||
Failure,
|
||||
failure,
|
||||
uiConnect,
|
||||
videoOptions,
|
||||
type Panel,
|
||||
} from "./common.js";
|
||||
|
||||
async function setup(store: DeviceStore, view?: string) {
|
||||
if (!process.stdin.isTTY || !process.stdout.isTTY)
|
||||
throw new Failure("SETUP_REQUIRES_TERMINAL");
|
||||
if (view) {
|
||||
const url = await pairBrowser(store.directory, view);
|
||||
if (!url) throw new Failure("PANEL_NOT_FOUND");
|
||||
if (await openURL(url)) process.stdout.write("浏览器配对入口已打开。\n");
|
||||
else
|
||||
process.stdout.write(
|
||||
`请在本机浏览器打开下方一次性链接（60 秒有效，不要发送到聊天）：\n${url}\n`,
|
||||
);
|
||||
return;
|
||||
}
|
||||
let hidden = false;
|
||||
const output = new Writable({
|
||||
write(chunk, _encoding, done) {
|
||||
if (!hidden) process.stdout.write(chunk);
|
||||
done();
|
||||
},
|
||||
});
|
||||
const rl = createInterface({
|
||||
input: process.stdin,
|
||||
output,
|
||||
terminal: true,
|
||||
historySize: 0,
|
||||
});
|
||||
const question = async (label: string) => (await rl.question(label)).trim();
|
||||
try {
|
||||
const config = await store.config();
|
||||
process.stdout.write(
|
||||
"CrossDesk MCP · 本地设备配置\n密码只保存到系统凭据库，不写入设备配置。\n",
|
||||
);
|
||||
const name = await question("设备别名：");
|
||||
const previous = config.devices.find((d) => d.alias === name);
|
||||
const device = previous?.device_id ?? (await question("9 位设备 ID："));
|
||||
const server =
|
||||
previous?.server ??
|
||||
((await question(`服务器 [${config.server}]：`)) || config.server);
|
||||
const port =
|
||||
previous?.port ??
|
||||
Number((await question(`端口 [${config.port}]：`)) || config.port);
|
||||
const permission = await question(
|
||||
"允许未来自动连接？[n / observe / control]：",
|
||||
);
|
||||
const environment = await question(
|
||||
"使用现有密码环境变量（留空使用系统凭据库）：",
|
||||
);
|
||||
let password: string | undefined;
|
||||
if (!environment) {
|
||||
hidden = true;
|
||||
process.stdout.write("访问密码（隐藏输入，可为空）：");
|
||||
try {
|
||||
password = await rl.question("");
|
||||
} finally {
|
||||
hidden = false;
|
||||
process.stdout.write("\n");
|
||||
}
|
||||
}
|
||||
parse(uiConnect, {
|
||||
device_id: device,
|
||||
label: name,
|
||||
server,
|
||||
port,
|
||||
password,
|
||||
});
|
||||
const d: Device = {
|
||||
alias: name,
|
||||
device_id: device,
|
||||
server,
|
||||
port,
|
||||
...(permission === "observe" || permission === "control"
|
||||
? { authorization: permission }
|
||||
: {}),
|
||||
...(environment
|
||||
? { credential: "environment", environment_variable: environment }
|
||||
: {}),
|
||||
};
|
||||
await store.save(d, password);
|
||||
password = undefined;
|
||||
process.stdout.write(
|
||||
"已保存。Agent 可调用 crossdesk_list_devices；自动连接仍需有效授权和凭据。\n",
|
||||
);
|
||||
} finally {
|
||||
hidden = false;
|
||||
rl.close();
|
||||
}
|
||||
}
|
||||
async function main() {
|
||||
const { values, positionals } = parseArgs({
|
||||
allowPositionals: true,
|
||||
strict: true,
|
||||
options: {
|
||||
stdio: { type: "boolean" },
|
||||
panel: { type: "string" },
|
||||
"video-path": { type: "string" },
|
||||
"video-decoder": { type: "string" },
|
||||
"config-dir": { type: "string" },
|
||||
"native-binary": { type: "string" },
|
||||
"open-panel": { type: "boolean" },
|
||||
view: { type: "string" },
|
||||
help: { type: "boolean" },
|
||||
version: { type: "boolean" },
|
||||
},
|
||||
});
|
||||
if (values.version) {
|
||||
process.stdout.write("0.1.0\n");
|
||||
return;
|
||||
}
|
||||
if (values.help || !positionals.length) {
|
||||
process.stdout.write(
|
||||
"crossdesk-mcp serve --stdio [--panel auto|apps|browser|off] [--video-path browser|minirtc] [--video-decoder auto|software] [--config-dir directory]\ncrossdesk-mcp setup [--config-dir directory]\ncrossdesk-mcp setup --open-panel --view VIEW_ID [--config-dir directory]\nDevelopment: --native-binary /absolute/path/to/crossdesk-cli\n",
|
||||
);
|
||||
return;
|
||||
}
|
||||
if (positionals.length !== 1 || !["serve", "setup"].includes(positionals[0]!))
|
||||
throw new Failure("INVALID_COMMAND");
|
||||
const store = new DeviceStore(
|
||||
values["config-dir"] ?? process.env.CROSSDESK_MCP_CONFIG_DIR,
|
||||
);
|
||||
await store.initialize();
|
||||
if (positionals[0] === "setup") {
|
||||
if (values["open-panel"] && !values.view)
|
||||
throw new Failure("VIEW_REQUIRED");
|
||||
await setup(store, values["open-panel"] ? values.view : undefined);
|
||||
return;
|
||||
}
|
||||
if (!values.stdio) throw new Failure("STDIO_REQUIRED");
|
||||
const config = await store.config();
|
||||
const policy: Panel = parse(
|
||||
panel,
|
||||
values.panel ?? config.panel,
|
||||
);
|
||||
const sessions = new Sessions(
|
||||
store,
|
||||
policy,
|
||||
values["native-binary"] ?? process.env.CROSSDESK_NATIVE_BINARY,
|
||||
parse(videoOptions, {
|
||||
decode_path: values["video-path"] ?? config.video.decode_path,
|
||||
decoder: values["video-decoder"] ?? config.video.decoder,
|
||||
}),
|
||||
);
|
||||
const browser = new BrowserPanel(sessions);
|
||||
const handle = serveStdio(() => createServer(sessions, browser), {
|
||||
transport: new StdioServerTransport(),
|
||||
onerror: () => process.stderr.write("crossdesk-mcp: protocol error\n"),
|
||||
});
|
||||
let closing = false;
|
||||
const close = async () => {
|
||||
if (closing) return;
|
||||
closing = true;
|
||||
await browser.close();
|
||||
await sessions.close();
|
||||
await handle.close();
|
||||
};
|
||||
process.stdin.once("end", () => {
|
||||
void close();
|
||||
});
|
||||
for (const signal of ["SIGINT", "SIGTERM", "SIGHUP"] as const)
|
||||
process.once(signal, () => {
|
||||
void close().finally(() => {
|
||||
process.exitCode = 0;
|
||||
});
|
||||
});
|
||||
process.once("uncaughtException", () => {
|
||||
process.stderr.write("crossdesk-mcp: INTERNAL_ERROR\n");
|
||||
void close().finally(() => {
|
||||
process.exitCode = 1;
|
||||
});
|
||||
});
|
||||
}
|
||||
main().catch((error) => {
|
||||
process.stderr.write(`crossdesk-mcp: ${failure(error).code}\n`);
|
||||
process.exitCode = 1;
|
||||
});
|
||||
@@ -0,0 +1,270 @@
|
||||
import { spawn, execFile, type ChildProcess } from "node:child_process";
|
||||
import { createRequire } from "node:module";
|
||||
import { dirname, join, resolve } from "node:path";
|
||||
import { promisify } from "node:util";
|
||||
import type { Writable } from "node:stream";
|
||||
import { Failure, type Data, type VideoOptions } from "./common.js";
|
||||
|
||||
export const NATIVE_VERSION = "0.2.0";
|
||||
const run = promisify(execFile);
|
||||
const require = createRequire(import.meta.url);
|
||||
function nativeEnvironment(): NodeJS.ProcessEnv {
|
||||
const env: NodeJS.ProcessEnv = {};
|
||||
for (const key of [
|
||||
"PATH",
|
||||
"HOME",
|
||||
"USERPROFILE",
|
||||
"SystemRoot",
|
||||
"SYSTEMROOT",
|
||||
"WINDIR",
|
||||
"TEMP",
|
||||
"TMP",
|
||||
"TMPDIR",
|
||||
"LANG",
|
||||
"LC_ALL",
|
||||
"SSL_CERT_FILE",
|
||||
"SSL_CERT_DIR",
|
||||
])
|
||||
if (process.env[key] !== undefined) env[key] = process.env[key];
|
||||
return env;
|
||||
}
|
||||
export async function nativeBinary(override?: string): Promise<string> {
|
||||
let binary: string;
|
||||
if (override) binary = resolve(override);
|
||||
else {
|
||||
const target = `${process.platform === "win32" ? "windows" : process.platform}-${process.arch}`;
|
||||
if (
|
||||
![
|
||||
"darwin-arm64",
|
||||
"darwin-x64",
|
||||
"linux-arm64",
|
||||
"linux-x64",
|
||||
"windows-x64",
|
||||
].includes(target)
|
||||
)
|
||||
throw new Failure("PLATFORM_UNSUPPORTED");
|
||||
if (
|
||||
process.platform === "linux" &&
|
||||
!(process.report.getReport() as Data).header.glibcVersionRuntime
|
||||
)
|
||||
throw new Failure("GLIBC_REQUIRED");
|
||||
try {
|
||||
const manifest = require.resolve(
|
||||
`@chunkdash/crossdesk-cli-${target}/package.json`,
|
||||
);
|
||||
if (require(manifest).version !== NATIVE_VERSION)
|
||||
throw new Failure("NATIVE_VERSION_MISMATCH");
|
||||
binary = join(
|
||||
dirname(manifest),
|
||||
"bin",
|
||||
process.platform === "win32" ? "crossdesk-cli.exe" : "crossdesk-cli",
|
||||
);
|
||||
} catch (error) {
|
||||
throw error instanceof Failure
|
||||
? error
|
||||
: new Failure("NATIVE_PACKAGE_MISSING");
|
||||
}
|
||||
}
|
||||
try {
|
||||
const { stdout } = await run(binary, ["doctor", "--json"], {
|
||||
timeout: 5000,
|
||||
maxBuffer: 65536,
|
||||
windowsHide: true,
|
||||
env: nativeEnvironment(),
|
||||
});
|
||||
const doctor = JSON.parse(stdout);
|
||||
if (doctor.automation_protocol !== "crossdesk-automation/1" ||
|
||||
doctor.automation_video_transport !== "loopback-websocket/1" ||
|
||||
!["h264-annexb/1", "nv12/1"].every((p) => doctor.automation_video_protocols?.includes(p)) ||
|
||||
!["auto", "software"].every((p) => doctor.automation_video_decoders?.includes(p)))
|
||||
throw new Failure("NATIVE_PROTOCOL_MISMATCH");
|
||||
} catch (error) {
|
||||
throw error instanceof Failure ? error : new Failure("NATIVE_START_FAILED");
|
||||
}
|
||||
return binary;
|
||||
}
|
||||
|
||||
type Pending = {
|
||||
resolve: (value: Data) => void;
|
||||
reject: (error: Failure) => void;
|
||||
clean: () => void;
|
||||
action: boolean;
|
||||
};
|
||||
export class NativeClient {
|
||||
private child: ChildProcess;
|
||||
private pending = new Map<number, Pending>();
|
||||
private sequence = 0;
|
||||
private buffer = "";
|
||||
private closed = false;
|
||||
private closing?: Promise<void>;
|
||||
constructor(binary: string, server: string, port: number, logs: string,
|
||||
readonly videoOptions: VideoOptions = { decode_path: "browser", decoder: "auto" }) {
|
||||
this.child = spawn(
|
||||
binary,
|
||||
[
|
||||
"agent",
|
||||
"--stdio",
|
||||
"--password-fd",
|
||||
"3",
|
||||
"--video-ws",
|
||||
"--server",
|
||||
server,
|
||||
"--port",
|
||||
String(port),
|
||||
"--log-dir",
|
||||
logs,
|
||||
],
|
||||
{
|
||||
stdio: ["pipe", "pipe", "ignore", "pipe"],
|
||||
shell: false,
|
||||
windowsHide: true,
|
||||
env: nativeEnvironment(),
|
||||
},
|
||||
);
|
||||
this.child.stdout!.setEncoding("utf8");
|
||||
this.child.stdout!.on("data", (part: string) => this.receive(part));
|
||||
this.child.stdin!.on("error", () => this.fail());
|
||||
this.child.stdio[3]!.on("error", () => this.fail());
|
||||
this.child.once("error", () => this.fail());
|
||||
this.child.once("exit", () => this.fail());
|
||||
}
|
||||
async connect(
|
||||
device: string,
|
||||
password: string,
|
||||
mode: string,
|
||||
display: number,
|
||||
signal?: AbortSignal,
|
||||
): Promise<Data> {
|
||||
if (Buffer.byteLength(password) > 256 || /[\0\r\n]/.test(password))
|
||||
throw new Failure("INVALID_CREDENTIAL");
|
||||
const hello = await this.request(
|
||||
"hello",
|
||||
{ protocol: "crossdesk-automation/1" },
|
||||
5000,
|
||||
signal,
|
||||
);
|
||||
if (hello.protocol !== "crossdesk-automation/1" ||
|
||||
hello.video_transport !== "loopback-websocket/1" ||
|
||||
!hello.video_protocols?.includes(this.videoOptions.decode_path === "minirtc" ? "nv12/1" : "h264-annexb/1") ||
|
||||
!hello.video_decoders?.includes(this.videoOptions.decoder))
|
||||
throw new Failure("NATIVE_PROTOCOL_MISMATCH");
|
||||
const secret = Buffer.from(password + "\n");
|
||||
password = "";
|
||||
(this.child.stdio[3] as Writable).end(secret, () => secret.fill(0));
|
||||
return this.request("connect", { device, mode, display, ...this.videoOptions }, 7000, signal);
|
||||
}
|
||||
request(
|
||||
method: string,
|
||||
args: Data = {},
|
||||
timeout = 7000,
|
||||
signal?: AbortSignal,
|
||||
): Promise<Data> {
|
||||
if (this.closed)
|
||||
return Promise.reject(new Failure("SESSION_CLOSED", method === "action"));
|
||||
if (signal?.aborted) return Promise.reject(new Failure("CANCELLED"));
|
||||
if (this.pending.size >= 32 || this.child.stdin!.writableLength > 262144)
|
||||
return Promise.reject(new Failure("BUSY", false, true));
|
||||
const id = ++this.sequence;
|
||||
const line = JSON.stringify({ ...args, id, method }) + "\n";
|
||||
if (Buffer.byteLength(line) > 65536)
|
||||
return Promise.reject(new Failure("MESSAGE_TOO_LARGE"));
|
||||
return new Promise((resolve, reject) => {
|
||||
const cancel = (code: string) => {
|
||||
const p = this.pending.get(id);
|
||||
if (!p) return;
|
||||
this.pending.delete(id);
|
||||
p.clean();
|
||||
// Cancellation is best effort; a timed-out action is never replayed.
|
||||
this.child.stdin?.write(
|
||||
JSON.stringify({
|
||||
id: ++this.sequence,
|
||||
method: "cancel",
|
||||
request_id: id,
|
||||
}) + "\n",
|
||||
);
|
||||
reject(new Failure(code, method === "action"));
|
||||
};
|
||||
const abort = () => cancel("CANCELLED");
|
||||
const timer = setTimeout(() => cancel("NATIVE_TIMEOUT"), timeout);
|
||||
signal?.addEventListener("abort", abort, { once: true });
|
||||
this.pending.set(id, {
|
||||
resolve,
|
||||
reject,
|
||||
action: method === "action",
|
||||
clean: () => {
|
||||
clearTimeout(timer);
|
||||
signal?.removeEventListener("abort", abort);
|
||||
},
|
||||
});
|
||||
this.child.stdin!.write(line);
|
||||
});
|
||||
}
|
||||
private receive(part: string) {
|
||||
this.buffer += part;
|
||||
if (Buffer.byteLength(this.buffer) > 8 * 1024 * 1024) {
|
||||
this.fail();
|
||||
this.child.kill();
|
||||
return;
|
||||
}
|
||||
for (;;) {
|
||||
const end = this.buffer.indexOf("\n");
|
||||
if (end < 0) break;
|
||||
const line = this.buffer.slice(0, end);
|
||||
this.buffer = this.buffer.slice(end + 1);
|
||||
try {
|
||||
if (Buffer.byteLength(line) > 4 * 1024 * 1024) throw new Error();
|
||||
const reply = JSON.parse(line);
|
||||
if (!Number.isSafeInteger(reply.id) || typeof reply.ok !== "boolean")
|
||||
throw new Error();
|
||||
const p = this.pending.get(reply.id);
|
||||
if (!p) continue;
|
||||
this.pending.delete(reply.id);
|
||||
p.clean();
|
||||
if (reply.ok) p.resolve(reply.result);
|
||||
else
|
||||
p.reject(
|
||||
new Failure(
|
||||
/^[A-Z_]+$/.test(reply.error?.code)
|
||||
? reply.error.code
|
||||
: "NATIVE_ERROR",
|
||||
!!reply.error?.side_effect_possible,
|
||||
!!reply.error?.retryable,
|
||||
),
|
||||
);
|
||||
} catch {
|
||||
this.fail();
|
||||
this.child.kill();
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
private fail() {
|
||||
this.closed = true;
|
||||
for (const p of this.pending.values()) {
|
||||
p.clean();
|
||||
p.reject(new Failure("SESSION_CLOSED", p.action));
|
||||
}
|
||||
this.pending.clear();
|
||||
}
|
||||
close(): Promise<void> {
|
||||
if (this.closing) return this.closing;
|
||||
this.closing = (async () => {
|
||||
if (!this.closed)
|
||||
await this.request("disconnect", {}, 1000).catch(() => {});
|
||||
this.child.stdin?.end();
|
||||
if (this.child.exitCode === null && this.child.signalCode === null)
|
||||
await new Promise<void>((resolve) => {
|
||||
const kill = setTimeout(() => {
|
||||
this.child.kill("SIGKILL");
|
||||
resolve();
|
||||
}, 2000);
|
||||
this.child.once("exit", () => {
|
||||
clearTimeout(kill);
|
||||
resolve();
|
||||
});
|
||||
});
|
||||
this.fail();
|
||||
})();
|
||||
return this.closing;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,399 @@
|
||||
import { readFile } from "node:fs/promises";
|
||||
import { z } from "zod";
|
||||
import { McpServer } from "@modelcontextprotocol/server";
|
||||
import type {
|
||||
CallToolResult,
|
||||
ClientCapabilities,
|
||||
Tool,
|
||||
ServerContext,
|
||||
} from "@modelcontextprotocol/server";
|
||||
import {
|
||||
getUiCapability,
|
||||
registerAppResource,
|
||||
registerAppTool,
|
||||
RESOURCE_MIME_TYPE,
|
||||
} from "@modelcontextprotocol/ext-apps/server";
|
||||
import { Sessions, type View } from "./sessions.js";
|
||||
import { BrowserPanel } from "./browser.js";
|
||||
import { uiAction, uiSchemas, type UIAction } from "./ui-actions.js";
|
||||
import {
|
||||
id,
|
||||
alias,
|
||||
mode,
|
||||
action,
|
||||
screenshot,
|
||||
Failure,
|
||||
failure,
|
||||
parse,
|
||||
token,
|
||||
type Data,
|
||||
} from "./common.js";
|
||||
|
||||
const resource = "ui://crossdesk/viewer.html";
|
||||
const uiNames: Record<UIAction, string> = {
|
||||
ready: "crossdesk_ui_ready",
|
||||
devices: "crossdesk_ui_list_devices",
|
||||
status: "crossdesk_ui_get_session",
|
||||
connect: "crossdesk_ui_connect",
|
||||
video_open: "crossdesk_video_open",
|
||||
control: "crossdesk_viewer_control",
|
||||
};
|
||||
function output(value: Data, meta?: Data): CallToolResult {
|
||||
const data = { ...value };
|
||||
const content: any[] = [];
|
||||
if (data.mime_type === "image/png" && typeof data.data === "string") {
|
||||
content.push({ type: "image", mimeType: data.mime_type, data: data.data });
|
||||
delete data.data;
|
||||
}
|
||||
if (data.observation?.data) {
|
||||
content.push({
|
||||
type: "image",
|
||||
mimeType: data.observation.mime_type,
|
||||
data: data.observation.data,
|
||||
});
|
||||
data.observation = { ...data.observation };
|
||||
delete data.observation.data;
|
||||
}
|
||||
content.unshift({ type: "text", text: JSON.stringify(data) });
|
||||
return { content, structuredContent: data, ...(meta ? { _meta: meta } : {}) };
|
||||
}
|
||||
export function createServer(
|
||||
sessions: Sessions,
|
||||
browser: BrowserPanel,
|
||||
): McpServer {
|
||||
const server = new McpServer(
|
||||
{ name: "crossdesk-mcp", version: "0.1.0" },
|
||||
{
|
||||
capabilities: { tools: {}, resources: {} },
|
||||
instructions:
|
||||
"Remote computer control. Open a panel for the user to select and authorize a device. Never request passwords in chat. When a remote task is pending, immediately follow crossdesk_open with crossdesk_get_session(view_id, wait_ms: 25000) in the same turn. Repeat bounded waits on wait_timed_out until ready or the user cancels; do not ask the user to type a connection confirmation or click Continue. On ready, continue the requested task within current authorization. On closed/failed, stop waiting and do not automatically reconnect. Obtain a screenshot before actions and pass its observation_id; coordinates refer to that returned image. Submitted actions are not remote execution acknowledgements. Never replay actions after uncertain outcomes. Human pause can only be resumed in the user panel.",
|
||||
},
|
||||
);
|
||||
type Definition = {
|
||||
name: string;
|
||||
schema: z.ZodType;
|
||||
description: string;
|
||||
ui?: boolean;
|
||||
app?: boolean;
|
||||
readOnly?: boolean;
|
||||
run: (p: Data, ctx: ServerContext) => Promise<CallToolResult>;
|
||||
};
|
||||
const definitions = new Map<string, Definition>();
|
||||
const supported = (ctx: ServerContext) => {
|
||||
const envelope = ctx.mcpReq.envelope as Data | undefined;
|
||||
const caps = envelope
|
||||
? envelope["io.modelcontextprotocol/clientCapabilities"]
|
||||
: server.server.getClientCapabilities();
|
||||
return (
|
||||
getUiCapability((caps ?? {}) as ClientCapabilities)?.mimeTypes?.includes(
|
||||
RESOURCE_MIME_TYPE,
|
||||
) ?? false
|
||||
);
|
||||
};
|
||||
const apps = (ctx: ServerContext) =>
|
||||
["auto", "apps"].includes(sessions.policy) && supported(ctx);
|
||||
const viewResult = (v: View, result: Data) =>
|
||||
output(
|
||||
result,
|
||||
v.panel === "apps"
|
||||
? {
|
||||
"crossdesk/ui": {
|
||||
view_id: v.id,
|
||||
view_generation: v.generation,
|
||||
ui_token: v.secret,
|
||||
},
|
||||
}
|
||||
: undefined,
|
||||
);
|
||||
const add = (
|
||||
name: string,
|
||||
schema: z.ZodType,
|
||||
description: string,
|
||||
run: Definition["run"],
|
||||
options: Partial<Definition> = {},
|
||||
) => {
|
||||
const d = { name, schema, description, run, ...options };
|
||||
definitions.set(name, d);
|
||||
const config = {
|
||||
description,
|
||||
inputSchema: schema,
|
||||
annotations: {
|
||||
readOnlyHint: !!d.readOnly,
|
||||
destructiveHint: d.name === "crossdesk_action",
|
||||
openWorldHint: true,
|
||||
},
|
||||
...(d.ui || d.app
|
||||
? {
|
||||
_meta: {
|
||||
ui: {
|
||||
...(d.ui ? { resourceUri: resource } : {}),
|
||||
visibility: d.app
|
||||
? ["app" as const]
|
||||
: ["model" as const, "app" as const],
|
||||
},
|
||||
},
|
||||
}
|
||||
: {}),
|
||||
};
|
||||
if (d.ui || d.app)
|
||||
registerAppTool(
|
||||
server,
|
||||
name,
|
||||
{ ...config, _meta: config._meta! },
|
||||
(p, ctx) => run(p as Data, ctx),
|
||||
);
|
||||
else server.registerTool(name, config, (p, ctx) => run(p as Data, ctx));
|
||||
};
|
||||
add(
|
||||
"crossdesk_open",
|
||||
z.strictObject({ device_alias: alias.optional() }),
|
||||
"Open CrossDesk so the user can choose a device, enter its password, and authorize this session. Returns immediately so the panel can render. To continue a pending remote task, call crossdesk_get_session with the returned view_id and wait_ms: 25000; no chat confirmation is needed.",
|
||||
async (p, ctx) => {
|
||||
if (p.device_alias) await sessions.store.device(p.device_alias);
|
||||
const v = sessions.create(
|
||||
sessions.choose(supported(ctx)),
|
||||
p.device_alias,
|
||||
);
|
||||
return viewResult(
|
||||
v,
|
||||
v.panel === "browser" ? await browser.open(v) : sessions.describe(v),
|
||||
);
|
||||
},
|
||||
{ ui: true },
|
||||
);
|
||||
add(
|
||||
"crossdesk_open_browser",
|
||||
z.strictObject({ view_id: id.optional() }),
|
||||
"Explicitly open the local browser panel after the user requests browser fallback. Does not create another remote connection.",
|
||||
async (p) => {
|
||||
if (!["auto", "browser"].includes(sessions.policy))
|
||||
throw new Failure("PANEL_POLICY_DENIED");
|
||||
const v = p.view_id
|
||||
? sessions.view(p.view_id)
|
||||
: sessions.create("browser");
|
||||
if (v.panel !== "browser") {
|
||||
await sessions.revokeVideo(v);
|
||||
v.panel = "browser";
|
||||
v.generation++;
|
||||
v.secret = token();
|
||||
v.ready = false;
|
||||
v.created = Date.now();
|
||||
}
|
||||
return output(await browser.open(v));
|
||||
},
|
||||
);
|
||||
add(
|
||||
"crossdesk_list_devices",
|
||||
z.strictObject({}),
|
||||
"List the local saved device book and persistent authorization, without credentials. This is not online discovery.",
|
||||
async () => output({ devices: await sessions.store.list() }),
|
||||
{ readOnly: true },
|
||||
);
|
||||
add(
|
||||
"crossdesk_connect",
|
||||
z.strictObject({
|
||||
device_alias: alias,
|
||||
mode: mode.default("observe"),
|
||||
display_index: z.int().min(0).max(63).default(0),
|
||||
}),
|
||||
"Connect only a saved, previously authorized device with available credentials. AUTH_REQUIRED means the user must authorize through the panel or setup.",
|
||||
async (p, ctx) => {
|
||||
const selected = sessions.choose(supported(ctx));
|
||||
const result = await sessions.connectSaved(
|
||||
p.device_alias,
|
||||
p.mode,
|
||||
p.display_index,
|
||||
ctx.mcpReq.signal,
|
||||
);
|
||||
const v = sessions.create(selected, p.device_alias);
|
||||
v.session = result.session_id;
|
||||
return viewResult(v, {
|
||||
...result,
|
||||
...(selected === "browser"
|
||||
? await browser.open(v)
|
||||
: sessions.describe(v)),
|
||||
state: result.state,
|
||||
});
|
||||
},
|
||||
{ ui: true },
|
||||
);
|
||||
add(
|
||||
"crossdesk_get_session",
|
||||
z.strictObject({
|
||||
session_id: id.optional(),
|
||||
view_id: id.optional(),
|
||||
wait_ms: z.int().min(0).max(25000).optional(),
|
||||
}),
|
||||
"Get connection, display geometry, frame age and control state. Supply exactly one of session_id or view_id. With view_id, wait for connection (default 25000 ms; 0 returns immediately). On wait_timed_out, repeat while the remote task is pending. On ready, continue without asking the user to send a message. Waiting never connects or grants control. session_id returns immediately and requires wait_ms to be 0 or omitted.",
|
||||
async (p, ctx) => {
|
||||
if (!!p.session_id === !!p.view_id)
|
||||
throw new Failure("SELECT_ONE_SESSION");
|
||||
if (p.session_id && p.wait_ms) throw new Failure("INVALID_ARGUMENT");
|
||||
return output(p.view_id
|
||||
? await sessions.waitForConnection(p.view_id, p.wait_ms ?? 25000, ctx.mcpReq.signal)
|
||||
: await sessions.status(p.session_id, undefined, ctx.mcpReq.signal));
|
||||
},
|
||||
{ readOnly: true },
|
||||
);
|
||||
add(
|
||||
"crossdesk_screenshot",
|
||||
screenshot,
|
||||
"Capture the remote display as a PNG image and immutable observation metadata. Coordinates in subsequent actions use this returned image. after_frame_id requires generation.",
|
||||
async (p, ctx) => output(await sessions.screenshot(p, ctx.mcpReq.signal)),
|
||||
{ readOnly: true },
|
||||
);
|
||||
add(
|
||||
"crossdesk_action",
|
||||
z.strictObject({
|
||||
session_id: id,
|
||||
operation_id: id,
|
||||
observation_id: id,
|
||||
action,
|
||||
observe_after: z.boolean().default(true),
|
||||
}),
|
||||
"Submit one structured action against a recent observation. Reuse operation_id only to retrieve that same operation, never replay uncertain effects with a new ID. Human pause prevents control. A post-action screenshot is not an execution acknowledgement.",
|
||||
async (p, ctx) => output(await sessions.action(p, ctx.mcpReq.signal)),
|
||||
);
|
||||
add(
|
||||
"crossdesk_set_control",
|
||||
z.strictObject({ session_id: id, enabled: z.boolean() }),
|
||||
"Enable or disable control within the user-authorized scope. Cannot override human pause or upgrade an observe-only connection.",
|
||||
async (p) => output(await sessions.control(p.session_id, p.enabled)),
|
||||
);
|
||||
add(
|
||||
"crossdesk_select_display",
|
||||
z.strictObject({ session_id: id, display_index: z.int().min(0).max(63) }),
|
||||
"Select a remote display. Invalidates previous observations; capture a new screenshot before any further input.",
|
||||
async (p) => output(await sessions.display(p.session_id, p.display_index)),
|
||||
);
|
||||
add(
|
||||
"crossdesk_disconnect",
|
||||
z.strictObject({ session_id: id }),
|
||||
"Disconnect the remote session, release input state and stop associated previews.",
|
||||
async (p) => output(await sessions.disconnect(p.session_id)),
|
||||
);
|
||||
for (const [name, schema] of Object.entries(uiSchemas)) {
|
||||
add(
|
||||
uiNames[name as UIAction],
|
||||
schema.extend({
|
||||
view_id: id,
|
||||
view_generation: z.int().min(1),
|
||||
ui_token: id,
|
||||
}),
|
||||
`CrossDesk panel only: ${name}.`,
|
||||
async (p, ctx) => {
|
||||
if (!apps(ctx)) throw new Failure("PANEL_UNSUPPORTED");
|
||||
const { view_id, view_generation, ui_token, ...args } = p;
|
||||
const v = sessions.authenticate(view_id, view_generation, ui_token);
|
||||
if (v.panel !== "apps") throw new Failure("VIEW_UNAUTHORIZED");
|
||||
// Video grants stay in the authenticated app-only response.
|
||||
const result = name === "video_open" ? await browser.video(v) : await uiAction(
|
||||
sessions,
|
||||
v,
|
||||
name,
|
||||
args,
|
||||
ctx.mcpReq.signal,
|
||||
);
|
||||
return {
|
||||
content: [
|
||||
{
|
||||
type: "text",
|
||||
text: name === "video_open" ? "Video viewer connection" : JSON.stringify(result),
|
||||
},
|
||||
],
|
||||
structuredContent: result,
|
||||
};
|
||||
},
|
||||
{ app: true },
|
||||
);
|
||||
}
|
||||
registerAppResource(
|
||||
server,
|
||||
"CrossDesk viewer",
|
||||
resource,
|
||||
{ mimeType: RESOURCE_MIME_TYPE },
|
||||
async (uri, ctx) => {
|
||||
if (!apps(ctx)) throw new Failure("PANEL_UNSUPPORTED");
|
||||
return {
|
||||
contents: [
|
||||
{
|
||||
uri: uri.href,
|
||||
mimeType: RESOURCE_MIME_TYPE,
|
||||
text: await readFile(
|
||||
new URL("./viewer-apps.html", import.meta.url),
|
||||
"utf8",
|
||||
),
|
||||
_meta: {
|
||||
ui: {
|
||||
prefersBorder: true,
|
||||
csp: { connectDomains: await browser.connectDomains(), resourceDomains: [] },
|
||||
},
|
||||
},
|
||||
},
|
||||
],
|
||||
};
|
||||
},
|
||||
);
|
||||
// Request-scoped projection avoids global tool mutations when modern clients
|
||||
// send concurrent calls with different capability envelopes.
|
||||
server.server.setRequestHandler("tools/list", async (_req, ctx) => ({
|
||||
tools: [...definitions.values()]
|
||||
.filter((d) => !d.app || apps(ctx))
|
||||
.map((d) => {
|
||||
const inputSchema = z.toJSONSchema(d.schema, {
|
||||
io: "input",
|
||||
}) as Tool["inputSchema"];
|
||||
if (d.name === "crossdesk_get_session")
|
||||
inputSchema.oneOf = [
|
||||
{ required: ["session_id"] },
|
||||
{ required: ["view_id"] },
|
||||
];
|
||||
return {
|
||||
name: d.name,
|
||||
description: d.description,
|
||||
inputSchema,
|
||||
annotations: {
|
||||
readOnlyHint: !!d.readOnly,
|
||||
destructiveHint: d.name === "crossdesk_action",
|
||||
openWorldHint: true,
|
||||
},
|
||||
...((d.ui || d.app) && apps(ctx)
|
||||
? {
|
||||
_meta: {
|
||||
ui: {
|
||||
...(d.ui ? { resourceUri: resource } : {}),
|
||||
visibility: d.app ? ["app"] : ["model", "app"],
|
||||
},
|
||||
...(d.ui ? { "ui/resourceUri": resource } : {}),
|
||||
},
|
||||
}
|
||||
: {}),
|
||||
};
|
||||
}),
|
||||
}));
|
||||
server.server.setRequestHandler("tools/call", async (req, ctx) => {
|
||||
try {
|
||||
const d = definitions.get(req.params.name);
|
||||
if (!d || (d.app && !apps(ctx))) throw new Failure("UNKNOWN_TOOL");
|
||||
if (Buffer.byteLength(JSON.stringify(req.params.arguments ?? {})) > 65536)
|
||||
throw new Failure("MESSAGE_TOO_LARGE");
|
||||
const p = parse(d.schema, req.params.arguments ?? {}) as Data;
|
||||
return await d.run(p, ctx);
|
||||
} catch (error) {
|
||||
const value = { error: failure(error) };
|
||||
return { ...output(value), isError: true };
|
||||
}
|
||||
});
|
||||
server.server.setRequestHandler("resources/list", async (_req, ctx) => ({
|
||||
resources: apps(ctx)
|
||||
? [
|
||||
{
|
||||
name: "CrossDesk viewer",
|
||||
uri: resource,
|
||||
mimeType: RESOURCE_MIME_TYPE,
|
||||
},
|
||||
]
|
||||
: [],
|
||||
}));
|
||||
return server;
|
||||
}
|
||||
@@ -0,0 +1,425 @@
|
||||
import { join } from "node:path";
|
||||
import { setTimeout as delay } from "node:timers/promises";
|
||||
import {
|
||||
Failure,
|
||||
token,
|
||||
equal,
|
||||
parse,
|
||||
uiConnect,
|
||||
type Data,
|
||||
type Panel,
|
||||
type VideoOptions,
|
||||
} from "./common.js";
|
||||
import { DeviceStore, type Device } from "./store.js";
|
||||
import { nativeBinary, NativeClient } from "./native.js";
|
||||
|
||||
export interface View {
|
||||
id: string;
|
||||
generation: number;
|
||||
secret: string;
|
||||
panel: Exclude<Panel, "auto">;
|
||||
created: number;
|
||||
ready: boolean;
|
||||
lastSeen: number;
|
||||
alias?: string;
|
||||
session?: string;
|
||||
disconnected?: boolean;
|
||||
}
|
||||
interface Session {
|
||||
id: string;
|
||||
device: Device;
|
||||
native: NativeClient;
|
||||
status: Data;
|
||||
release: () => Promise<void>;
|
||||
}
|
||||
export class Sessions {
|
||||
readonly views = new Map<string, View>();
|
||||
private current?: Session;
|
||||
private connecting = false;
|
||||
private closing = false;
|
||||
private binary?: Promise<string>;
|
||||
constructor(
|
||||
readonly store: DeviceStore,
|
||||
readonly policy: Panel,
|
||||
private nativePath?: string,
|
||||
private videoOptions?: VideoOptions,
|
||||
) {}
|
||||
choose(apps: boolean): View["panel"] {
|
||||
if (this.policy === "apps" && !apps) throw new Failure("PANEL_UNSUPPORTED");
|
||||
return this.policy === "auto" ? (apps ? "apps" : "browser") : this.policy;
|
||||
}
|
||||
create(panel: View["panel"], alias?: string): View {
|
||||
for (const [id, view] of this.views)
|
||||
if (Date.now() - view.lastSeen > 30 * 60 * 1000) {
|
||||
void this.revokeVideo(view);
|
||||
this.views.delete(id);
|
||||
}
|
||||
if (this.views.size >= 32) throw new Failure("VIEW_LIMIT");
|
||||
const view: View = {
|
||||
id: token(),
|
||||
generation: 1,
|
||||
secret: token(),
|
||||
panel,
|
||||
created: Date.now(),
|
||||
lastSeen: Date.now(),
|
||||
ready: false,
|
||||
alias,
|
||||
session: this.current?.id,
|
||||
};
|
||||
this.views.set(view.id, view);
|
||||
return view;
|
||||
}
|
||||
view(id: string): View {
|
||||
const v = this.views.get(id);
|
||||
if (!v) throw new Failure("VIEW_NOT_FOUND");
|
||||
return v;
|
||||
}
|
||||
authenticate(id: string, generation: number, secret: string): View {
|
||||
const v = this.view(id);
|
||||
if (v.generation !== generation || !equal(v.secret, secret))
|
||||
throw new Failure("VIEW_UNAUTHORIZED");
|
||||
return v;
|
||||
}
|
||||
describe(v: View): Data {
|
||||
return {
|
||||
view_id: v.id,
|
||||
view_generation: v.generation,
|
||||
panel: v.panel,
|
||||
panel_status: v.ready
|
||||
? "ready"
|
||||
: Date.now() - v.created > 10000
|
||||
? "not_confirmed"
|
||||
: "waiting",
|
||||
session_id: v.session ?? null,
|
||||
state: v.session ? "session_available" : v.disconnected ? "closed" : "awaiting_user",
|
||||
device_alias: v.alias ?? null,
|
||||
next_step: v.session
|
||||
? "Call crossdesk_get_session with view_id."
|
||||
: v.disconnected
|
||||
? "The session was disconnected. Do not automatically reconnect."
|
||||
: v.panel === "off"
|
||||
? "Use setup to configure credentials and authorization, then crossdesk_connect with the saved device alias. Never send passwords in chat."
|
||||
: "The user selects and authorizes a device in the panel. To continue a pending remote task, immediately call crossdesk_get_session with this view_id and wait_ms: 25000; repeat on wait_timed_out. Do not ask the user to send a connection confirmation message.",
|
||||
};
|
||||
}
|
||||
private session(id: string): Session {
|
||||
if (!this.current || this.current.id !== id)
|
||||
throw new Failure("SESSION_NOT_FOUND");
|
||||
return this.current;
|
||||
}
|
||||
async status(sessionId?: string, viewId?: string, signal?: AbortSignal): Promise<Data> {
|
||||
const view = viewId ? this.view(viewId) : undefined;
|
||||
const id = sessionId ?? view?.session;
|
||||
if (!id && view) return this.describe(view);
|
||||
const s = this.session(id!);
|
||||
// A panel connection may still be performing the native hello handshake.
|
||||
// Reading native status here would race initialization and could close it.
|
||||
if (this.connecting) return {
|
||||
...(view ? this.describe(view) : {}),
|
||||
session_id: s.id,
|
||||
device_alias: s.device.alias,
|
||||
state: "connecting",
|
||||
control: false,
|
||||
};
|
||||
try {
|
||||
s.status = await s.native.request("status", {}, 5000, signal);
|
||||
} catch {
|
||||
if (signal?.aborted) throw new Failure("CANCELLED");
|
||||
s.status = { state: "closed", control: false };
|
||||
}
|
||||
// The user may have disconnected or replaced the session during the read.
|
||||
if (this.current !== s || (view && view.session !== id))
|
||||
return view ? this.describe(view) : { session_id: id, state: "closed", control: false };
|
||||
if (["closed", "failed"].includes(s.status.state)) {
|
||||
await s.native.close();
|
||||
await s.release();
|
||||
}
|
||||
return {
|
||||
...s.status,
|
||||
session_id: s.id,
|
||||
device_alias: s.device.alias,
|
||||
...(view ? this.describe(view) : {}),
|
||||
state: s.status.state,
|
||||
...(s.status.state === "ready" ? {
|
||||
next_step: "Connection ready. Continue the user's pending task within the current authorization. Obtain a screenshot before input; respect human_paused and control. No user confirmation message is needed.",
|
||||
} : ["closed", "failed", "disconnected"].includes(s.status.state) ? {
|
||||
next_step: "The connection ended. Stop waiting and do not automatically reconnect or replay actions.",
|
||||
} : {}),
|
||||
};
|
||||
}
|
||||
async waitForConnection(viewId: string, waitMs: number, signal?: AbortSignal): Promise<Data> {
|
||||
const deadline = Date.now() + waitMs;
|
||||
let result = this.describe(this.view(viewId));
|
||||
const timedOut = () => ({
|
||||
...result,
|
||||
wait_timed_out: true,
|
||||
next_step: "Still waiting for the user to connect in the panel. Call crossdesk_get_session again with the same view_id and wait_ms: 25000 while the remote task is pending. Do not ask the user to type a connection confirmation.",
|
||||
});
|
||||
const timeout = new AbortController();
|
||||
const timer = waitMs > 0 ? setTimeout(() => timeout.abort(), waitMs) : undefined;
|
||||
const waitSignal = signal ? AbortSignal.any([signal, timeout.signal]) : timeout.signal;
|
||||
try {
|
||||
for (;;) {
|
||||
if (this.closing || signal?.aborted) throw new Failure("CANCELLED");
|
||||
const view = this.view(viewId);
|
||||
view.lastSeen = Date.now();
|
||||
result = await this.status(undefined, viewId, waitSignal);
|
||||
if (waitMs === 0 || view.panel === "off" ||
|
||||
["ready", "closed", "failed", "disconnected"].includes(result.state))
|
||||
return result;
|
||||
const remaining = deadline - Date.now();
|
||||
if (remaining <= 0) return timedOut();
|
||||
await delay(Math.min(250, remaining), undefined, { signal: waitSignal });
|
||||
}
|
||||
} catch (error) {
|
||||
if (this.closing || signal?.aborted) throw new Failure("CANCELLED");
|
||||
if (timeout.signal.aborted) return timedOut();
|
||||
throw error;
|
||||
} finally {
|
||||
clearTimeout(timer);
|
||||
}
|
||||
}
|
||||
async connectSaved(
|
||||
name: string,
|
||||
mode: string,
|
||||
display: number,
|
||||
signal?: AbortSignal,
|
||||
): Promise<Data> {
|
||||
const d = await this.store.device(name);
|
||||
if (
|
||||
!d.authorization ||
|
||||
(mode === "control" && d.authorization !== "control")
|
||||
)
|
||||
throw new Failure("AUTH_REQUIRED");
|
||||
return this.connect(
|
||||
d,
|
||||
await this.store.password(d),
|
||||
mode,
|
||||
display,
|
||||
undefined,
|
||||
signal,
|
||||
);
|
||||
}
|
||||
async connectUI(v: View, args: unknown, signal?: AbortSignal): Promise<Data> {
|
||||
const p = parse(uiConnect, args);
|
||||
if (!!p.device_alias === !!p.device_id)
|
||||
throw new Failure("SELECT_ONE_DEVICE");
|
||||
const config = await this.store.config();
|
||||
const d: Device = p.device_alias
|
||||
? { ...(await this.store.device(p.device_alias)) }
|
||||
: {
|
||||
alias: p.label ?? p.device_id!,
|
||||
device_id: p.device_id!,
|
||||
server: p.server ?? config.server,
|
||||
port: p.port ?? config.port,
|
||||
};
|
||||
if (p.device_alias && (p.server || p.port || p.label))
|
||||
throw new Failure("INVALID_ARGUMENT");
|
||||
if ((p.remember_password || p.persistent_authorization) && !p.save_device)
|
||||
throw new Failure("SAVE_DEVICE_REQUIRED");
|
||||
let password = p.password ?? (await this.store.password(d));
|
||||
p.password = undefined;
|
||||
// Persist only after successful authentication, avoiding poisoned saved passwords.
|
||||
const result = await this.connect(
|
||||
d,
|
||||
password,
|
||||
p.mode,
|
||||
p.display_index,
|
||||
v,
|
||||
signal,
|
||||
);
|
||||
if (p.save_device) {
|
||||
d.authorization = p.persistent_authorization ? p.mode : undefined;
|
||||
try {
|
||||
await this.store.save(d, p.remember_password ? password : undefined);
|
||||
} catch (error) {
|
||||
result.persistence_error =
|
||||
error instanceof Failure ? error.code : "CONFIG_WRITE_FAILED";
|
||||
}
|
||||
}
|
||||
password = "";
|
||||
return result;
|
||||
}
|
||||
private async connect(
|
||||
d: Device,
|
||||
password: string,
|
||||
mode: string,
|
||||
display: number,
|
||||
view?: View,
|
||||
signal?: AbortSignal,
|
||||
): Promise<Data> {
|
||||
if (this.closing) throw new Failure("SESSION_CLOSED");
|
||||
if (this.connecting) throw new Failure("BUSY");
|
||||
if (
|
||||
this.current &&
|
||||
!["closed", "failed", "disconnected"].includes(
|
||||
(await this.status(this.current.id)).state,
|
||||
)
|
||||
)
|
||||
throw new Failure("SESSION_LIMIT");
|
||||
this.connecting = true;
|
||||
let release: (() => Promise<void>) | undefined;
|
||||
let native: NativeClient | undefined;
|
||||
try {
|
||||
if (this.current) await this.disconnect(this.current.id);
|
||||
release = await this.store.lock(
|
||||
`device:${d.server}:${d.port}:${d.device_id}`,
|
||||
);
|
||||
this.binary ??= nativeBinary(this.nativePath).catch((e) => {
|
||||
this.binary = undefined;
|
||||
throw e;
|
||||
});
|
||||
const binary = await this.binary;
|
||||
if (this.closing || signal?.aborted) throw new Failure("CANCELLED");
|
||||
native = new NativeClient(
|
||||
binary,
|
||||
d.server,
|
||||
d.port,
|
||||
join(this.store.directory, "logs"),
|
||||
this.videoOptions ?? (await this.store.config()).video,
|
||||
);
|
||||
const s: Session = {
|
||||
id: token(),
|
||||
device: d,
|
||||
native,
|
||||
status: {},
|
||||
release,
|
||||
};
|
||||
this.current = s;
|
||||
if (view) {
|
||||
view.session = s.id;
|
||||
view.alias = d.alias;
|
||||
view.disconnected = false;
|
||||
}
|
||||
s.status = await native.connect(
|
||||
d.device_id,
|
||||
password,
|
||||
mode,
|
||||
display,
|
||||
signal,
|
||||
);
|
||||
password = "";
|
||||
const deadline = Date.now() + 35000;
|
||||
while (Date.now() < deadline) {
|
||||
if (this.closing || signal?.aborted) throw new Failure("CANCELLED");
|
||||
s.status = await native.request("status", {}, 5000, signal);
|
||||
if (s.status.state === "ready" && s.status.frame_age_ms <= 2000)
|
||||
return { ...s.status, session_id: s.id, device_alias: s.device.alias };
|
||||
if (["failed", "closed", "disconnected"].includes(s.status.state))
|
||||
throw new Failure(
|
||||
s.status.error === "incorrect_password"
|
||||
? "AUTH_FAILED"
|
||||
: "CONNECTION_FAILED",
|
||||
);
|
||||
await delay(100, undefined, { signal });
|
||||
}
|
||||
throw new Failure("CONNECTION_TIMEOUT");
|
||||
} catch (error) {
|
||||
await native?.close();
|
||||
await release?.();
|
||||
if (this.current?.native === native) this.current = undefined;
|
||||
if (view) view.session = undefined;
|
||||
throw error;
|
||||
} finally {
|
||||
password = "";
|
||||
this.connecting = false;
|
||||
}
|
||||
}
|
||||
async screenshot(p: Data, signal?: AbortSignal): Promise<Data> {
|
||||
const { session_id, ...args } = p;
|
||||
if (args.after_frame_id !== undefined && args.generation === undefined)
|
||||
throw new Failure("GENERATION_REQUIRED");
|
||||
return this.session(session_id).native.request(
|
||||
"screenshot",
|
||||
args,
|
||||
(args.timeout_ms ?? 2000) + 3000,
|
||||
signal,
|
||||
);
|
||||
}
|
||||
async action(p: Data, signal?: AbortSignal): Promise<Data> {
|
||||
const { session_id, observe_after = true, ...args } = p;
|
||||
const s = this.session(session_id);
|
||||
const result = await s.native.request("action", args, 5000, signal);
|
||||
if (observe_after && result.status === "submitted") {
|
||||
try {
|
||||
result.observation = await this.screenshot(
|
||||
{
|
||||
session_id,
|
||||
generation: result.generation,
|
||||
after_frame_id: result.after_frame_id,
|
||||
timeout_ms: 2000,
|
||||
},
|
||||
signal,
|
||||
);
|
||||
} catch (error) {
|
||||
result.observation_error =
|
||||
error instanceof Failure ? error.code : "OBSERVATION_FAILED";
|
||||
}
|
||||
}
|
||||
return result;
|
||||
}
|
||||
async control(id: string, enabled: boolean) {
|
||||
return this.session(id).native.request("control", { enabled });
|
||||
}
|
||||
async display(id: string, index: number) {
|
||||
return this.session(id).native.request("display", { index });
|
||||
}
|
||||
async disconnect(id: string) {
|
||||
const s = this.session(id);
|
||||
await s.native.close();
|
||||
await s.release();
|
||||
if (this.current === s) this.current = undefined;
|
||||
for (const v of this.views.values())
|
||||
if (v.session === id) {
|
||||
v.session = undefined;
|
||||
v.disconnected = true;
|
||||
}
|
||||
return { session_id: id, state: "closed" };
|
||||
}
|
||||
async viewerControl(v: View, operation: string) {
|
||||
if (!v.session) throw new Failure("NOT_READY");
|
||||
if (operation === "disconnect") return this.disconnect(v.session);
|
||||
return this.session(v.session).native.request("viewer_control", {
|
||||
operation,
|
||||
});
|
||||
}
|
||||
async openVideo(v: View, origin: string): Promise<Data> {
|
||||
if (!v.session || this.views.get(v.id) !== v) throw new Failure("NOT_READY");
|
||||
const s = this.session(v.session);
|
||||
if (this.closing || this.connecting || ["closed", "failed"].includes(s.status.state))
|
||||
throw new Failure("NOT_READY");
|
||||
const generation = v.generation, session = v.session;
|
||||
const ticket = token();
|
||||
const result = await s.native.request("video_open", {
|
||||
ticket, view: v.id, view_generation: generation, session, origin, apps: v.panel === "apps",
|
||||
});
|
||||
if (this.current !== s || this.views.get(v.id) !== v ||
|
||||
v.generation !== generation || v.session !== session) {
|
||||
await s.native.request("video_revoke", { view: v.id }).catch(() => {});
|
||||
throw new Failure("SESSION_CHANGED");
|
||||
}
|
||||
if (result.transport !== "loopback-websocket/1" ||
|
||||
typeof result.url !== "string" || !/^ws:\/\/127\.0\.0\.1:\d+\/api\/video$/.test(result.url) ||
|
||||
result.protocol !== (s.native.videoOptions.decode_path === "minirtc" ? "nv12/1" : "h264-annexb/1") ||
|
||||
result.decoder !== s.native.videoOptions.decoder)
|
||||
throw new Failure("NATIVE_PROTOCOL_MISMATCH");
|
||||
return { ...result, ticket };
|
||||
}
|
||||
async revokeVideo(view?: View) {
|
||||
const s = this.current;
|
||||
if (!s || (view && view.session !== s.id)) return;
|
||||
try {
|
||||
await s.native.request("video_revoke", view ? { view: view.id } : {});
|
||||
} catch {
|
||||
// Losing the control channel must not leave an authorized video socket
|
||||
// alive after its view was removed or replaced.
|
||||
if (this.current === s) await this.disconnect(s.id);
|
||||
else await s.native.close();
|
||||
}
|
||||
}
|
||||
async close() {
|
||||
this.closing = true;
|
||||
if (this.current) await this.disconnect(this.current.id);
|
||||
this.views.clear();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,191 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import {
|
||||
mkdir,
|
||||
readFile,
|
||||
writeFile,
|
||||
rename,
|
||||
unlink,
|
||||
open,
|
||||
chmod,
|
||||
} from "node:fs/promises";
|
||||
import { homedir } from "node:os";
|
||||
import { join, resolve } from "node:path";
|
||||
import { z } from "zod";
|
||||
import {
|
||||
alias,
|
||||
host,
|
||||
mode,
|
||||
panel,
|
||||
videoOptions,
|
||||
Failure,
|
||||
parse,
|
||||
token,
|
||||
type Data,
|
||||
} from "./common.js";
|
||||
|
||||
const deviceSchema = z.strictObject({
|
||||
alias,
|
||||
device_id: z.string().regex(/^\d{9}$/),
|
||||
server: host,
|
||||
port: z.int().min(1).max(65535),
|
||||
authorization: mode.optional(),
|
||||
credential: z.enum(["keyring", "environment"]).optional(),
|
||||
environment_variable: z
|
||||
.string()
|
||||
.regex(/^[A-Z][A-Z0-9_]{0,127}$/)
|
||||
.optional(),
|
||||
});
|
||||
export type Device = z.infer<typeof deviceSchema>;
|
||||
const configSchema = z.strictObject({
|
||||
version: z.literal(1),
|
||||
panel: panel.default("auto"),
|
||||
video: videoOptions.prefault({}),
|
||||
server: host.default("api.crossdesk.cn"),
|
||||
port: z.int().min(1).max(65535).default(9099),
|
||||
devices: z.array(deviceSchema).max(256),
|
||||
});
|
||||
export type Config = z.infer<typeof configSchema>;
|
||||
export class DeviceStore {
|
||||
readonly directory: string;
|
||||
readonly file: string;
|
||||
constructor(directory?: string) {
|
||||
this.directory = resolve(
|
||||
directory ?? join(homedir(), ".config", "crossdesk", "mcp"),
|
||||
);
|
||||
this.file = join(this.directory, "devices.json");
|
||||
}
|
||||
async initialize() {
|
||||
await mkdir(this.directory, { recursive: true, mode: 0o700 });
|
||||
if (process.platform !== "win32") await chmod(this.directory, 0o700);
|
||||
await mkdir(join(this.directory, "logs"), { recursive: true, mode: 0o700 });
|
||||
}
|
||||
async config(): Promise<Config> {
|
||||
try {
|
||||
return parse(configSchema, JSON.parse(await readFile(this.file, "utf8")));
|
||||
} catch (error) {
|
||||
if ((error as NodeJS.ErrnoException).code === "ENOENT")
|
||||
return configSchema.parse({ version: 1, devices: [] });
|
||||
throw new Failure("CONFIG_INVALID");
|
||||
}
|
||||
}
|
||||
async list(): Promise<Data[]> {
|
||||
return (await this.config()).devices.map((d) => ({
|
||||
device_alias: d.alias,
|
||||
device_id: d.device_id,
|
||||
server: d.server,
|
||||
port: d.port,
|
||||
authorization: d.authorization ?? "none",
|
||||
credential_provider: d.credential ?? "none",
|
||||
}));
|
||||
}
|
||||
async device(name: string): Promise<Device> {
|
||||
const d = (await this.config()).devices.find((d) => d.alias === name);
|
||||
if (!d) throw new Failure("DEVICE_NOT_FOUND");
|
||||
return d;
|
||||
}
|
||||
private account(d: Device) {
|
||||
return createHash("sha256")
|
||||
.update(JSON.stringify([this.directory, d.server, d.port, d.device_id]))
|
||||
.digest("hex");
|
||||
}
|
||||
private async entry(d: Device) {
|
||||
try {
|
||||
const { AsyncEntry } = await import("@napi-rs/keyring");
|
||||
return new AsyncEntry("crossdesk-mcp", this.account(d), {
|
||||
linux: { store: "secret-service" },
|
||||
});
|
||||
} catch {
|
||||
throw new Failure("CREDENTIAL_STORE_UNAVAILABLE");
|
||||
}
|
||||
}
|
||||
async password(d: Device): Promise<string> {
|
||||
if (d.credential === "environment") {
|
||||
const secret =
|
||||
d.environment_variable && process.env[d.environment_variable];
|
||||
if (secret !== undefined) return secret;
|
||||
} else if (d.credential === "keyring") {
|
||||
try {
|
||||
const secret = await (await this.entry(d)).getPassword();
|
||||
if (secret !== null) return secret;
|
||||
} catch {
|
||||
throw new Failure("CREDENTIAL_STORE_UNAVAILABLE");
|
||||
}
|
||||
}
|
||||
throw new Failure("AUTH_REQUIRED");
|
||||
}
|
||||
async save(d: Device, password?: string) {
|
||||
parse(deviceSchema, d);
|
||||
const lock = await this.lock("config");
|
||||
try {
|
||||
const config = await this.config();
|
||||
const previous = config.devices.find((item) => item.alias === d.alias);
|
||||
if (
|
||||
previous &&
|
||||
(previous.device_id !== d.device_id ||
|
||||
previous.server !== d.server ||
|
||||
previous.port !== d.port)
|
||||
)
|
||||
throw new Failure("DEVICE_ALIAS_CONFLICT");
|
||||
if (password !== undefined) {
|
||||
try {
|
||||
await (await this.entry(d)).setPassword(password);
|
||||
d.credential = "keyring";
|
||||
delete d.environment_variable;
|
||||
} catch {
|
||||
throw new Failure("CREDENTIAL_STORE_UNAVAILABLE");
|
||||
}
|
||||
}
|
||||
config.devices = [
|
||||
...config.devices.filter((item) => item.alias !== d.alias),
|
||||
d,
|
||||
];
|
||||
parse(configSchema, config);
|
||||
const temporary = `${this.file}.${token()}.tmp`;
|
||||
try {
|
||||
await writeFile(temporary, JSON.stringify(config, null, 2) + "\n", {
|
||||
mode: 0o600,
|
||||
flag: "wx",
|
||||
});
|
||||
await rename(temporary, this.file);
|
||||
} finally {
|
||||
await unlink(temporary).catch(() => {});
|
||||
}
|
||||
} finally {
|
||||
await lock();
|
||||
}
|
||||
}
|
||||
async lock(name: string): Promise<() => Promise<void>> {
|
||||
const path = join(
|
||||
this.directory,
|
||||
`lock-${createHash("sha256").update(name).digest("hex").slice(0, 24)}`,
|
||||
);
|
||||
for (let attempt = 0; attempt < 2; ++attempt) {
|
||||
try {
|
||||
const handle = await open(path, "wx", 0o600);
|
||||
await handle.writeFile(String(process.pid));
|
||||
await handle.close();
|
||||
let released = false;
|
||||
return async () => {
|
||||
if (released) return;
|
||||
released = true;
|
||||
await unlink(path).catch(() => {});
|
||||
};
|
||||
} catch (error) {
|
||||
if ((error as NodeJS.ErrnoException).code !== "EEXIST")
|
||||
throw new Failure("CONFIG_WRITE_FAILED");
|
||||
const pid = Number(await readFile(path, "utf8").catch(() => "0"));
|
||||
if (!Number.isSafeInteger(pid) || pid < 1)
|
||||
throw new Failure("DEVICE_BUSY");
|
||||
try {
|
||||
process.kill(pid, 0);
|
||||
throw new Failure("DEVICE_BUSY");
|
||||
} catch (e) {
|
||||
if ((e as NodeJS.ErrnoException).code !== "ESRCH")
|
||||
throw new Failure("DEVICE_BUSY");
|
||||
}
|
||||
await unlink(path).catch(() => {});
|
||||
}
|
||||
}
|
||||
throw new Failure("DEVICE_BUSY");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
import { z } from "zod";
|
||||
import { parse, uiConnect, Failure, type Data } from "./common.js";
|
||||
import { Sessions, type View } from "./sessions.js";
|
||||
|
||||
export const uiSchemas = {
|
||||
ready: z.strictObject({}),
|
||||
devices: z.strictObject({}),
|
||||
status: z.strictObject({}),
|
||||
connect: uiConnect,
|
||||
video_open: z.strictObject({}),
|
||||
control: z.strictObject({
|
||||
operation: z.enum(["pause", "resume", "disconnect"]),
|
||||
}),
|
||||
};
|
||||
export type UIAction = keyof typeof uiSchemas;
|
||||
export async function uiAction(
|
||||
sessions: Sessions,
|
||||
view: View,
|
||||
name: string,
|
||||
args: unknown,
|
||||
signal?: AbortSignal,
|
||||
): Promise<Data> {
|
||||
if (!Object.hasOwn(uiSchemas, name)) throw new Failure("UNKNOWN_ACTION");
|
||||
const p = parse(uiSchemas[name as UIAction], args) as Data;
|
||||
view.lastSeen = Date.now();
|
||||
switch (name) {
|
||||
case "ready":
|
||||
view.ready = true;
|
||||
return sessions.describe(view);
|
||||
case "devices": {
|
||||
const config = await sessions.store.config();
|
||||
return {
|
||||
devices: await sessions.store.list(),
|
||||
defaults: { server: config.server, port: config.port },
|
||||
};
|
||||
}
|
||||
case "status":
|
||||
return sessions.status(undefined, view.id);
|
||||
case "connect":
|
||||
return sessions.connectUI(view, p, signal);
|
||||
case "control":
|
||||
return sessions.viewerControl(view, p.operation);
|
||||
default:
|
||||
throw new Failure("UNKNOWN_ACTION");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2022",
|
||||
"module": "NodeNext",
|
||||
"moduleResolution": "NodeNext",
|
||||
"strict": true,
|
||||
"noUncheckedIndexedAccess": true,
|
||||
"noFallthroughCasesInSwitch": true,
|
||||
"outDir": "dist",
|
||||
"rootDir": "src",
|
||||
"skipLibCheck": true,
|
||||
"lib": ["ES2022", "DOM"]
|
||||
},
|
||||
"include": ["src/**/*.ts"]
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2022",
|
||||
"module": "ESNext",
|
||||
"moduleResolution": "Bundler",
|
||||
"strict": true,
|
||||
"noUncheckedIndexedAccess": true,
|
||||
"noEmit": true,
|
||||
"skipLibCheck": true,
|
||||
"lib": ["ES2022", "DOM", "DOM.Iterable"],
|
||||
"types": []
|
||||
},
|
||||
"include": ["ui/**/*.ts"]
|
||||
}
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 7.5 KiB |
@@ -0,0 +1,230 @@
|
||||
type FrameInfo = { width: number; height: number; fps: number; frames: number };
|
||||
function parameters(data: Uint8Array) {
|
||||
let codec = "", signature = "";
|
||||
for (let i = 0; i + 4 < data.length; i++) {
|
||||
if (data[i] || data[i + 1]) continue;
|
||||
const prefix = data[i + 2] === 1 ? 3 : data[i + 2] === 0 && data[i + 3] === 1 ? 4 : 0;
|
||||
if (!prefix) continue;
|
||||
const start = i + prefix, type = data[start]! & 31;
|
||||
if (type !== 7 && type !== 8) continue;
|
||||
let end = start + 1;
|
||||
while (end + 3 < data.length && !(data[end] === 0 && data[end + 1] === 0 &&
|
||||
(data[end + 2] === 1 || (data[end + 2] === 0 && data[end + 3] === 1)))) end++;
|
||||
if (end + 3 >= data.length) end = data.length;
|
||||
if (end - start > 65536) throw new Error("VIDEO_INVALID");
|
||||
const hex = [...data.subarray(start, end)].map((n) => n.toString(16).padStart(2, "0")).join("");
|
||||
signature += hex;
|
||||
if (type === 7 && end >= start + 4) codec = "avc1." + hex.slice(2, 8);
|
||||
i = end - 1;
|
||||
}
|
||||
return { codec, signature };
|
||||
}
|
||||
|
||||
export class VideoPlayer {
|
||||
private socket?: WebSocket;
|
||||
private decoder?: VideoDecoder;
|
||||
private stopped = false;
|
||||
private failed = false;
|
||||
private waitingKey = true;
|
||||
private generation = -1;
|
||||
private sequence = -1;
|
||||
private signature = "";
|
||||
private queue: ArrayBuffer[] = [];
|
||||
private queuedBytes = 0;
|
||||
private pumping = false;
|
||||
private keyRequested = 0;
|
||||
private firstFrame?: ReturnType<typeof setTimeout>;
|
||||
private latest?: VideoFrame;
|
||||
private drawing = 0;
|
||||
private count = 0;
|
||||
private samples = 0;
|
||||
private sampledAt = performance.now();
|
||||
private fps = 0;
|
||||
private raw = false;
|
||||
private software = false;
|
||||
private automatic = true;
|
||||
constructor(private canvas: HTMLCanvasElement,
|
||||
private onFrame: (info: FrameInfo) => void,
|
||||
private onError: (error: Error) => void) {}
|
||||
|
||||
start(url: string, ticket: string, protocol = "h264-annexb/1", decoder = "auto") {
|
||||
if (!["h264-annexb/1", "nv12/1"].includes(protocol) ||
|
||||
!["auto", "software"].includes(decoder)) throw new Error("VIDEO_INVALID");
|
||||
this.raw = protocol === "nv12/1";
|
||||
this.automatic = decoder === "auto";
|
||||
this.software = !this.automatic;
|
||||
if (typeof VideoFrame === "undefined" || (!this.raw && typeof VideoDecoder === "undefined"))
|
||||
throw new Error("VIDEO_UNSUPPORTED");
|
||||
if (!/^ws:\/\/127\.0\.0\.1:\d+\/api\/video$/.test(url)) throw new Error("VIDEO_INVALID");
|
||||
const socket = this.socket = new WebSocket(url);
|
||||
socket.binaryType = "arraybuffer";
|
||||
socket.onopen = () => socket.send(JSON.stringify({ ticket }));
|
||||
socket.onerror = () => this.fail(new Error("VIDEO_UNAVAILABLE"));
|
||||
socket.onclose = () => { if (!this.stopped) this.fail(new Error("VIDEO_UNAVAILABLE")); };
|
||||
this.firstFrame = setTimeout(() => this.fail(new Error("VIDEO_UNAVAILABLE")), 10000);
|
||||
socket.onmessage = (event) => {
|
||||
if (this.stopped || !(event.data instanceof ArrayBuffer)) return;
|
||||
const data = event.data;
|
||||
if (data.byteLength < (this.raw ? 38 : 25) ||
|
||||
data.byteLength > (this.raw ? 16777216 * 3 / 2 + 32 : 4 * 1024 * 1024 + 24))
|
||||
return this.fail(new Error("VIDEO_INVALID"));
|
||||
const header = new DataView(data);
|
||||
if (header.getUint32(0) !== (this.raw ? 0x43444e31 : 0x43445631))
|
||||
return this.fail(new Error("VIDEO_INVALID"));
|
||||
// Receipt, not presentation: bounds native writes even when the decoder
|
||||
// or the canvas is slow. This message goes directly to the native CLI.
|
||||
if (socket.readyState === WebSocket.OPEN) socket.send(JSON.stringify({
|
||||
type: "ack", generation: header.getUint32(4), sequence: header.getUint32(8),
|
||||
}));
|
||||
if (this.raw || this.queue.length >= 8 || this.queuedBytes + data.byteLength > 8 * 1024 * 1024) {
|
||||
this.queue = [];
|
||||
this.queuedBytes = 0;
|
||||
this.waitingKey = true;
|
||||
this.requestKey();
|
||||
}
|
||||
this.queue.push(data);
|
||||
this.queuedBytes += data.byteLength;
|
||||
void this.pump();
|
||||
};
|
||||
}
|
||||
private requestKey() {
|
||||
if (this.raw) return;
|
||||
if (Date.now() - this.keyRequested < 1000 || this.socket?.readyState !== WebSocket.OPEN) return;
|
||||
this.keyRequested = Date.now();
|
||||
this.socket.send(JSON.stringify({ type: "keyframe" }));
|
||||
}
|
||||
private async pump() {
|
||||
if (this.pumping) return;
|
||||
this.pumping = true;
|
||||
try {
|
||||
while (!this.stopped && this.queue.length) {
|
||||
const bytes = this.queue.shift()!;
|
||||
this.queuedBytes -= bytes.byteLength;
|
||||
const packet = new DataView(bytes);
|
||||
if (packet.getUint32(0) !== (this.raw ? 0x43444e31 : 0x43445631) || packet.getUint32(20) > 1)
|
||||
throw new Error("VIDEO_INVALID");
|
||||
const generation = packet.getUint32(4), sequence = packet.getUint32(8);
|
||||
const key = packet.getUint32(20) === 1;
|
||||
if (generation < this.generation) continue;
|
||||
const timestamp = Number(packet.getBigUint64(12));
|
||||
if (!Number.isSafeInteger(timestamp)) throw new Error("VIDEO_INVALID");
|
||||
if (this.raw) {
|
||||
const width = packet.getUint32(24), height = packet.getUint32(28);
|
||||
if (!key || !width || !height || (width & 1) || (height & 1) ||
|
||||
width > 8192 || height > 8192 || width * height > 16777216 ||
|
||||
bytes.byteLength !== 32 + width * height * 3 / 2)
|
||||
throw new Error("VIDEO_INVALID");
|
||||
if (generation === this.generation && sequence <= this.sequence) continue;
|
||||
this.generation = generation;
|
||||
this.sequence = sequence;
|
||||
// Match the native NV12 screenshot conversion's limited-range BT.601.
|
||||
this.present(new VideoFrame(new Uint8Array(bytes, 32), {
|
||||
format: "NV12", codedWidth: width, codedHeight: height, timestamp,
|
||||
colorSpace: { matrix: "smpte170m", primaries: "smpte170m", transfer: "smpte170m", fullRange: false },
|
||||
transfer: [bytes],
|
||||
} as VideoFrameBufferInit & { transfer: ArrayBuffer[] }));
|
||||
continue;
|
||||
}
|
||||
if (generation !== this.generation || (this.sequence >= 0 && sequence !== this.sequence + 1))
|
||||
this.waitingKey = true;
|
||||
this.generation = generation;
|
||||
this.sequence = sequence;
|
||||
if ((this.decoder?.decodeQueueSize ?? 0) >= 4) this.waitingKey = true;
|
||||
if (this.waitingKey && !key) { this.requestKey(); continue; }
|
||||
const data = new Uint8Array(bytes, 24);
|
||||
if (key) {
|
||||
const { codec, signature } = parameters(data);
|
||||
if (!codec) { this.waitingKey = true; this.requestKey(); continue; }
|
||||
if (!this.decoder || this.waitingKey || signature !== this.signature) {
|
||||
const config: VideoDecoderConfig = { codec, optimizeForLatency: true,
|
||||
hardwareAcceleration: this.software ? "prefer-software" : "prefer-hardware" };
|
||||
if (!(await VideoDecoder.isConfigSupported(config)).supported) {
|
||||
if (this.software) throw new Error("VIDEO_UNSUPPORTED");
|
||||
this.software = true;
|
||||
config.hardwareAcceleration = "prefer-software";
|
||||
if (!(await VideoDecoder.isConfigSupported(config)).supported) throw new Error("VIDEO_UNSUPPORTED");
|
||||
}
|
||||
if (this.stopped) break;
|
||||
if (this.decoder && this.decoder.state !== "closed") this.decoder.close();
|
||||
const decoder = new VideoDecoder({
|
||||
output: (frame) => {
|
||||
if (this.stopped || this.decoder !== decoder) { frame.close(); return; }
|
||||
this.present(frame);
|
||||
},
|
||||
error: () => {
|
||||
if (this.stopped || this.decoder !== decoder) return;
|
||||
this.decoder = undefined;
|
||||
if (this.software || !this.automatic) {
|
||||
this.fail(new Error("VIDEO_UNAVAILABLE"));
|
||||
return;
|
||||
}
|
||||
this.software = true;
|
||||
this.waitingKey = true;
|
||||
this.requestKey();
|
||||
},
|
||||
});
|
||||
this.decoder = decoder;
|
||||
decoder.configure(config);
|
||||
this.signature = signature;
|
||||
}
|
||||
this.waitingKey = false;
|
||||
}
|
||||
this.decoder?.decode(new EncodedVideoChunk({ type: key ? "key" : "delta", timestamp, data }));
|
||||
}
|
||||
} catch (error) {
|
||||
this.fail(error instanceof Error ? error : new Error("VIDEO_INVALID"));
|
||||
} finally { this.pumping = false; }
|
||||
}
|
||||
private present(frame: VideoFrame) {
|
||||
this.latest?.close();
|
||||
this.latest = frame;
|
||||
if (!this.drawing) this.drawing = requestAnimationFrame(() => this.draw());
|
||||
}
|
||||
private draw() {
|
||||
this.drawing = 0;
|
||||
const frame = this.latest;
|
||||
this.latest = undefined;
|
||||
if (!frame) return;
|
||||
try {
|
||||
if (this.stopped) return;
|
||||
const width = frame.displayWidth, height = frame.displayHeight;
|
||||
if (width > 8192 || height > 8192 || width * height > 16777216) throw new Error("VIDEO_INVALID");
|
||||
if (this.canvas.width !== width || this.canvas.height !== height) {
|
||||
this.canvas.width = width;
|
||||
this.canvas.height = height;
|
||||
}
|
||||
this.canvas.getContext("2d", { alpha: false })!.drawImage(frame, 0, 0);
|
||||
clearTimeout(this.firstFrame);
|
||||
this.count++;
|
||||
this.samples++;
|
||||
const elapsed = performance.now() - this.sampledAt;
|
||||
if (elapsed >= 1000) {
|
||||
this.fps = this.samples * 1000 / elapsed;
|
||||
this.samples = 0;
|
||||
this.sampledAt = performance.now();
|
||||
}
|
||||
this.canvas.dataset.frames = String(this.count);
|
||||
this.onFrame({ width, height, frames: this.count, fps: this.fps });
|
||||
} catch { this.fail(new Error("VIDEO_INVALID")); }
|
||||
finally { frame.close(); }
|
||||
}
|
||||
private fail(error: Error) {
|
||||
if (this.failed || this.stopped) return;
|
||||
this.failed = true;
|
||||
this.stop();
|
||||
this.onError(error);
|
||||
}
|
||||
stop() {
|
||||
this.stopped = true;
|
||||
clearTimeout(this.firstFrame);
|
||||
this.socket?.close();
|
||||
if (this.decoder && this.decoder.state !== "closed") this.decoder.close();
|
||||
this.decoder = undefined;
|
||||
cancelAnimationFrame(this.drawing);
|
||||
this.drawing = 0;
|
||||
this.latest?.close();
|
||||
this.latest = undefined;
|
||||
this.queue = [];
|
||||
this.queuedBytes = 0;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,184 @@
|
||||
:root {
|
||||
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "PingFang SC", "Microsoft YaHei", sans-serif;
|
||||
color: #292d33;
|
||||
background: #fff;
|
||||
font-size: 13px;
|
||||
color-scheme: light;
|
||||
--accent: #3269b7;
|
||||
--muted: #777d85;
|
||||
--line: #e4e6e9;
|
||||
}
|
||||
* { box-sizing: border-box; }
|
||||
html, body { width: 100%; height: 100%; margin: 0; }
|
||||
body { overflow: hidden; }
|
||||
[hidden] { display: none !important; }
|
||||
button, input, select { font: inherit; }
|
||||
button, summary, input, select { -webkit-tap-highlight-color: transparent; }
|
||||
:focus-visible { outline: 2px solid #7194c6; outline-offset: 2px; }
|
||||
#app { width: 100%; height: 100vh; height: 100dvh; display: flex; flex-direction: column; }
|
||||
.app-header { display: flex; align-items: center; justify-content: space-between; flex-wrap: wrap; gap: 8px 16px; padding: 12px 20px; border-bottom: 1px solid var(--line); flex: none; }
|
||||
.brand, .header-right { display: flex; align-items: center; gap: 10px; min-width: 0; }
|
||||
.brand-logo { display: block; width: 28px; height: 28px; }
|
||||
h1 { margin: 0; font-size: 16px; font-weight: 600; line-height: 1.5; white-space: nowrap; }
|
||||
h2 { margin: 0; font-size: 13px; font-weight: 600; }
|
||||
.connection { color: var(--muted); font-size: 12px; white-space: nowrap; }
|
||||
.connection[data-state="online"] { color: #397557; }
|
||||
.connection[data-state="connecting"] { color: #8a682d; }
|
||||
.connection[data-state="offline"] { color: #a15149; }
|
||||
#details-toggle { display: none; }
|
||||
main { display: grid; grid-template-columns: 270px minmax(0, 1fr); flex: 1; min-height: 0; }
|
||||
aside { min-width: 0; min-height: 0; overflow: auto; overscroll-behavior: contain; padding: 22px 20px; border-right: 1px solid var(--line); }
|
||||
#sidebar-title { margin-bottom: 20px; }
|
||||
fieldset { border: 0; margin: 0; padding: 0; min-width: 0; }
|
||||
label { display: block; margin: 16px 0 7px; font-size: 12px; }
|
||||
fieldset > label:first-child { margin-top: 0; }
|
||||
.optional { color: var(--muted); margin-left: 4px; font-size: 11px; }
|
||||
input:not([type="checkbox"]), select { display: block; width: 100%; height: 36px; min-width: 0; border: 1px solid #d7dade; border-radius: 4px; padding: 7px 9px; background: #fff; color: inherit; }
|
||||
input:not([type="checkbox"]):focus, select:focus { border-color: var(--accent); outline: 1px solid var(--accent); }
|
||||
input:disabled, select:disabled { background: #f5f5f5; color: #999; }
|
||||
input::placeholder { color: #92969c; }
|
||||
#device-id, #session-device-id { font-variant-numeric: tabular-nums; }
|
||||
.password-field { position: relative; }
|
||||
.password-field input { padding-right: 48px; }
|
||||
.password-field button { position: absolute; right: 2px; top: 2px; height: 32px; min-height: 32px; padding: 0 8px; border: 0; background: transparent; color: var(--muted); }
|
||||
.field-hint, .privacy, .session-note { color: var(--muted); font-size: 11px; line-height: 1.7; margin: 7px 0 0; }
|
||||
.preferences { border-top: 1px solid var(--line); margin-top: 20px; padding-top: 7px; }
|
||||
details { font-size: 12px; }
|
||||
summary { padding: 9px 0; cursor: pointer; color: #666d76; }
|
||||
summary::marker { font-size: 10px; }
|
||||
.checks { padding-bottom: 10px; }
|
||||
.checks label { display: flex; align-items: flex-start; gap: 8px; margin-top: 13px; font-size: 12px; line-height: 1.5; cursor: pointer; }
|
||||
.checks label:has(input:disabled) { color: #999; cursor: default; }
|
||||
.checks small { display: block; margin-top: 2px; color: var(--muted); font-size: 11px; }
|
||||
input[type="checkbox"] { width: 14px; height: 14px; flex: none; margin: 2px 0 0; accent-color: var(--accent); }
|
||||
.row { display: flex; gap: 12px; }
|
||||
.row > div { flex: 1; min-width: 0; }
|
||||
button { display: inline-flex; align-items: center; justify-content: center; min-height: 32px; padding: 6px 10px; border: 1px solid #d9dce0; border-radius: 4px; background: #fff; color: #4d535c; font-size: 12px; cursor: pointer; white-space: nowrap; }
|
||||
button:not(:disabled):hover { background: #f3f4f5; border-color: #bfc4cb; }
|
||||
button:not(:disabled):active { background: #e9ebee; }
|
||||
button:disabled { cursor: default; color: #a3a7ad; background: #fafafa; border-color: #e7e9eb; }
|
||||
.primary { width: 100%; min-height: 36px; background: var(--accent); color: #fff; border-color: var(--accent); margin-top: 20px; }
|
||||
.primary:not(:disabled):hover { background: #285b9f; border-color: #285b9f; }
|
||||
.privacy { margin-top: 12px; }
|
||||
#error { flex: none; max-height: 25%; overflow: auto; padding: 10px 20px; background: #fff5f2; border-bottom: 1px solid #edddd6; color: #964a37; font-size: 12px; line-height: 1.6; overflow-wrap: anywhere; }
|
||||
.device-card strong { display: block; font-size: 14px; font-weight: 500; overflow-wrap: anywhere; }
|
||||
.device-card > span { display: block; margin-top: 5px; color: var(--muted); font-size: 12px; }
|
||||
.session-details { margin: 20px 0 0; font-size: 12px; }
|
||||
.session-details > div { display: flex; justify-content: space-between; gap: 12px; padding: 11px 0; border-bottom: 1px solid #eee; }
|
||||
.session-details dt { color: var(--muted); flex: none; }
|
||||
.session-details dd { margin: 0; text-align: right; overflow-wrap: anywhere; }
|
||||
#session-agent[data-paused="true"] { color: #956e2d; }
|
||||
.session-note { margin-top: 18px; }
|
||||
.workspace { display: flex; flex-direction: column; gap: 10px; min-width: 0; min-height: 0; overflow: auto; padding: 16px; background: #f6f7f8; }
|
||||
.viewer-card { display: flex; flex-direction: column; flex: 1; min-height: 160px; min-width: 0; background: #fff; border: 1px solid var(--line); border-radius: 5px; overflow: hidden; }
|
||||
.viewer-toolbar { display: flex; justify-content: space-between; align-items: center; flex-wrap: wrap; gap: 8px 12px; padding: 9px 12px; flex: none; }
|
||||
.viewer-title { display: flex; align-items: center; gap: 10px; min-width: 0; flex: 1; }
|
||||
.viewer-title h2 { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
|
||||
#video-state { color: var(--muted); font-size: 11px; white-space: nowrap; flex: none; }
|
||||
#video-state[data-state="stale"] { color: #956e2d; }
|
||||
.viewer-controls { display: flex; gap: 6px; flex-wrap: wrap; }
|
||||
.danger:not(:disabled) { color: #9a4e47; }
|
||||
.danger:not(:disabled):hover { border-color: #d6b4ae; background: #fff6f4; }
|
||||
/* The screen takes the remaining panel height; object-fit preserves the remote aspect ratio. */
|
||||
.screen { position: relative; flex: 1; min-height: 60px; width: 100%; display: flex; align-items: center; justify-content: center; overflow: hidden; isolation: isolate; border-radius: 8px; background: #17191c; }
|
||||
.screen canvas { position: absolute; inset: 0; width: 100%; height: 100%; object-fit: contain; }
|
||||
/* Overlay the control indicator so toggling control never resizes the video. */
|
||||
.control-halo { display: none; }
|
||||
#app[data-controlling="true"] .screen::before,
|
||||
#app[data-controlling="true"] .screen::after { content: ""; position: absolute; inset: 0; border-radius: inherit; pointer-events: none; }
|
||||
#app[data-controlling="true"] .screen::before { z-index: 1; box-shadow: inset 0 0 10px #3976ff40, inset 0 0 3px #8ccaff70; }
|
||||
#app[data-controlling="true"] .control-halo {
|
||||
--halo-spread: 14px;
|
||||
display: block;
|
||||
position: absolute;
|
||||
inset: 0;
|
||||
z-index: 2;
|
||||
overflow: hidden;
|
||||
border-radius: inherit;
|
||||
pointer-events: none;
|
||||
/* Fade the light inward at the edges, leaving the desktop itself clear. */
|
||||
mask-image:
|
||||
linear-gradient(to bottom, #000, #0007 4px, transparent var(--halo-spread), transparent calc(100% - var(--halo-spread)), #0007 calc(100% - 4px), #000),
|
||||
linear-gradient(to right, #000, #0007 4px, transparent var(--halo-spread), transparent calc(100% - var(--halo-spread)), #0007 calc(100% - 4px), #000);
|
||||
}
|
||||
.control-halo::before {
|
||||
content: "";
|
||||
position: absolute;
|
||||
inset: -22%;
|
||||
background:
|
||||
radial-gradient(ellipse 32% 38% at 24% 26%, #d5f3ff 0%, #76c4ff 20%, #427bff80 50%, transparent 100%),
|
||||
radial-gradient(ellipse 38% 40% at 78% 75%, #c4d7ff 0%, #637cff 24%, #436bff70 60%, transparent 100%),
|
||||
linear-gradient(125deg, #3c8aff70, #687dff50 55%, #4ab5ff80);
|
||||
filter: blur(8px);
|
||||
opacity: .9;
|
||||
/* Move a cached light layer rather than repainting an animated gradient. */
|
||||
animation: control-halo-drift 9s ease-in-out infinite alternate;
|
||||
will-change: transform, opacity;
|
||||
}
|
||||
#app[data-controlling="true"] .screen::after {
|
||||
z-index: 3;
|
||||
padding: 1px;
|
||||
background: linear-gradient(125deg, #dbf6ffb0, #5e98ff70 30%, #c2e9ffc0 53%, #7e9dff80 75%, #b0e6ffb0);
|
||||
-webkit-mask: linear-gradient(#fff 0 0) content-box, linear-gradient(#fff 0 0);
|
||||
-webkit-mask-composite: xor;
|
||||
mask: linear-gradient(#fff 0 0) content-box, linear-gradient(#fff 0 0);
|
||||
mask-composite: exclude;
|
||||
}
|
||||
@keyframes control-halo-drift {
|
||||
0% { transform: translate3d(-7%, -5%, 0) scale(1); opacity: .7; }
|
||||
50% { opacity: 1; }
|
||||
100% { transform: translate3d(7%, 5%, 0) scale(1.05); opacity: .85; }
|
||||
}
|
||||
@media (prefers-reduced-motion: reduce) {
|
||||
.control-halo::before { animation: none; will-change: auto; }
|
||||
}
|
||||
#empty { text-align: center; padding: 16px; max-width: 100%; color: #b4b7bd; }
|
||||
#empty h3 { font-size: 14px; font-weight: 400; margin: 0 0 8px; }
|
||||
#empty p { color: #7e848e; font-size: 12px; line-height: 1.7; margin: 0; }
|
||||
.stale { position: absolute; left: 12px; bottom: 12px; padding: 7px 10px; background: #25272cdd; color: #e2c495; border-radius: 3px; font-size: 11px; }
|
||||
.telemetry { display: flex; align-items: center; flex-wrap: wrap; gap: 7px 18px; padding: 9px 12px; flex: none; color: var(--muted); font-size: 11px; }
|
||||
.stream-metrics { display: flex; gap: 14px; font-variant-numeric: tabular-nums; }
|
||||
#permission { margin-left: auto; }
|
||||
#permission[data-state="paused"] { color: #956e2d; }
|
||||
.panel-status { flex: none; font-size: 12px; line-height: 1.6; color: var(--muted); margin: 0; }
|
||||
.video-options { display: flex; flex-wrap: wrap; gap: 8px; flex: none; }
|
||||
.video-options:not(:has(button:not([hidden]))) { display: none; }
|
||||
.focus-mode main { grid-template-columns: minmax(0, 1fr); }
|
||||
.focus-mode .app-header { display: none; }
|
||||
.focus-mode aside, .focus-mode #details-toggle { display: none !important; }
|
||||
@media (max-width: 760px) {
|
||||
.app-header { padding: 10px 14px; }
|
||||
main { display: flex; flex-direction: column; }
|
||||
aside { flex: 1; border-right: 0; padding: 20px; }
|
||||
#app:not([data-connected="true"]) .workspace { display: none; }
|
||||
#app[data-connected="true"] aside { display: none; flex: none; max-height: 40%; border-bottom: 1px solid var(--line); }
|
||||
#app[data-connected="true"].details-open aside { display: block; }
|
||||
#app[data-connected="true"] #details-toggle { display: inline-flex; }
|
||||
#app[data-connected="true"] .workspace { flex: 1; }
|
||||
.workspace { padding: 10px; }
|
||||
#sidebar-title { margin-bottom: 16px; }
|
||||
.session-details { margin-top: 12px; }
|
||||
}
|
||||
@media (max-width: 480px) {
|
||||
#app[data-controlling="true"] .control-halo { --halo-spread: 10px; }
|
||||
.app-header { gap: 8px; padding: 10px 12px; }
|
||||
.brand, .header-right { gap: 7px; }
|
||||
.brand-logo { width: 24px; height: 24px; }
|
||||
h1 { font-size: 15px; }
|
||||
.connection { font-size: 11px; }
|
||||
#details-toggle { padding: 5px 7px; min-height: 28px; font-size: 11px; }
|
||||
.workspace { padding: 8px; gap: 8px; }
|
||||
.viewer-toolbar { padding: 9px; }
|
||||
.viewer-title, .viewer-controls { width: 100%; flex: auto; }
|
||||
.viewer-controls button { flex: 1; min-height: 36px; }
|
||||
.telemetry { padding: 8px 9px; gap: 6px 10px; }
|
||||
#session-info { display: none; }
|
||||
.stream-metrics { gap: 10px; }
|
||||
#permission { font-size: 10px; }
|
||||
}
|
||||
@media (max-height: 460px) and (min-width: 481px) {
|
||||
.app-header { padding-top: 6px; padding-bottom: 6px; }
|
||||
.workspace { padding: 8px; gap: 6px; }
|
||||
.viewer-toolbar { padding-top: 5px; padding-bottom: 5px; }
|
||||
.telemetry { padding-top: 6px; padding-bottom: 6px; }
|
||||
}
|
||||
@@ -0,0 +1,546 @@
|
||||
import type { App } from "@modelcontextprotocol/ext-apps";
|
||||
import "./viewer.css";
|
||||
import { VideoPlayer } from "./video-player.js";
|
||||
|
||||
declare const CROSSDESK_APPS: boolean;
|
||||
declare const CROSSDESK_LOGO: string;
|
||||
type Data = Record<string, any>;
|
||||
type Invoke = (name: string, args?: Data) => Promise<Data>;
|
||||
const root = document.querySelector<HTMLDivElement>("#app")!;
|
||||
root.innerHTML = `
|
||||
<header class="app-header">
|
||||
<div class="brand"><img class="brand-logo" src="${CROSSDESK_LOGO}" alt="" width="28" height="28"><h1>CrossDesk MCP</h1></div>
|
||||
<div class="header-right"><button id="details-toggle" aria-expanded="false" aria-controls="sidebar">连接信息</button><span class="connection" id="connection" role="status"><span id="connection-label">等待连接</span></span></div>
|
||||
</header>
|
||||
<div id="error" role="alert" hidden></div>
|
||||
<main>
|
||||
<aside id="sidebar" aria-labelledby="sidebar-title">
|
||||
<h2 id="sidebar-title">连接设备</h2>
|
||||
<div id="connect-form" role="form" aria-label="连接远程电脑">
|
||||
<fieldset id="connection-fields">
|
||||
<label for="device">设备</label><select id="device"><option value="">连接新设备</option></select>
|
||||
<div id="new-device">
|
||||
<label for="device-id">设备 ID</label><input id="device-id" inputmode="numeric" autocomplete="off" placeholder="输入 9 位设备 ID" maxlength="11">
|
||||
</div>
|
||||
<label for="password">访问密码</label><div class="password-field"><input id="password" type="password" autocomplete="off" maxlength="256" placeholder="输入远端访问密码"><button type="button" id="password-toggle" aria-label="显示密码" aria-pressed="false">显示</button></div>
|
||||
<label for="mode">授权范围</label><select id="mode" aria-describedby="mode-description"><option value="observe">仅查看画面</option><option value="control">允许 Agent 操作</option></select>
|
||||
<p class="field-hint" id="mode-description">Agent 可以查看屏幕，无法操作键鼠。</p>
|
||||
<details class="preferences"><summary>保存与授权</summary>
|
||||
<div id="new-device-label"><label for="label">设备备注 <span class="optional">可选</span></label><input id="label" placeholder="例如：工作电脑" maxlength="80"></div>
|
||||
<div class="checks">
|
||||
<label><input id="save" type="checkbox"><span>保存设备</span></label>
|
||||
<label><input id="remember" type="checkbox" disabled><span>记住密码<small>保存在本机系统凭据库</small></span></label>
|
||||
<label><input id="authorize" type="checkbox" disabled><span>允许下次自动连接<small>沿用本次授权范围</small></span></label>
|
||||
</div>
|
||||
</details>
|
||||
<details id="advanced"><summary>高级设置</summary>
|
||||
<label for="server">服务器</label><input id="server" value="api.crossdesk.cn" autocomplete="off">
|
||||
<div class="row"><div><label for="port">端口</label><input id="port" type="number" min="1" max="65535" value="9099"></div><div><label for="display">显示器索引</label><input id="display" type="number" min="0" max="63" value="0"></div></div>
|
||||
</details>
|
||||
</fieldset>
|
||||
<button id="connect" type="button" class="primary" disabled><span id="connect-label">连接</span></button>
|
||||
<p class="privacy">密码不会发送到 Agent 对话中。</p>
|
||||
</div>
|
||||
<div id="session-card" hidden>
|
||||
<div class="device-card"><strong id="session-device">远程电脑</strong><span id="session-platform">已连接</span></div>
|
||||
<dl class="session-details"><div id="session-device-id-row" hidden><dt>设备 ID</dt><dd id="session-device-id"></dd></div><div><dt>访问权限</dt><dd id="session-access"></dd></div><div><dt>显示器</dt><dd id="session-display"></dd></div><div><dt>Agent 状态</dt><dd id="session-agent"></dd></div></dl>
|
||||
<p class="session-note">关闭页面不会断开连接。</p>
|
||||
</div>
|
||||
</aside>
|
||||
<section class="workspace" aria-label="远程画面与会话控制">
|
||||
<div class="viewer-card">
|
||||
<div class="viewer-toolbar">
|
||||
<div class="viewer-title"><h2 id="device-name">远程画面</h2><span id="video-state">待连接</span></div>
|
||||
<div class="viewer-controls"><button id="pause" disabled><span id="pause-label">暂停控制</span></button><button id="disconnect" class="danger" disabled>断开连接</button><button id="focus-view" aria-label="放大画面" aria-pressed="false" disabled>放大画面</button></div>
|
||||
</div>
|
||||
<div class="screen" id="screen">
|
||||
<canvas id="frame" role="img" aria-label="远程桌面实时画面" hidden></canvas>
|
||||
<div class="control-halo" aria-hidden="true"></div>
|
||||
<div id="empty"><h3 id="empty-title">未连接设备</h3><p id="empty-description">选择设备并输入访问密码。</p></div>
|
||||
<div class="stale" id="stale" role="status" hidden>画面暂未更新</div>
|
||||
</div>
|
||||
<div class="telemetry"><div class="stream-metrics"><span id="resolution">—</span><span id="fps">—</span></div><span id="session-info">等待设备连接</span><span id="permission">等待授权</span></div>
|
||||
</div>
|
||||
<div class="video-options"><button id="video-retry" hidden>重新加载画面</button><button id="video-browser" hidden>在浏览器中观看</button></div>
|
||||
<p id="panel-status" class="panel-status" role="status" hidden></p>
|
||||
</section>
|
||||
</main>`;
|
||||
const element = <T extends HTMLElement = HTMLElement>(id: string) =>
|
||||
document.getElementById(id) as T;
|
||||
const input = (id: string) => element<HTMLInputElement>(id);
|
||||
const select = (id: string) => element<HTMLSelectElement>(id);
|
||||
let invoke: Invoke;
|
||||
let app: App | undefined;
|
||||
let context: Data | undefined;
|
||||
let status: Data = {};
|
||||
let savedDevices: Data[] = [];
|
||||
let controlBusy = false;
|
||||
let disconnectedByUser = false;
|
||||
let frameSize = "", frameRate = "";
|
||||
let player: VideoPlayer | undefined;
|
||||
let videoStarting = false, videoFailed = false, videoEpoch = 0;
|
||||
let videoSession: string | undefined;
|
||||
let videoGeneration: number | undefined;
|
||||
let imageAt = 0;
|
||||
let busy = false,
|
||||
polling = false,
|
||||
started = false;
|
||||
const messages: Record<string, string> = {
|
||||
VIDEO_UNSUPPORTED: "当前浏览器不支持此视频格式，请使用支持 WebCodecs 的浏览器。",
|
||||
VIDEO_UNAVAILABLE: "视频通道暂不可用。可重新加载画面；内嵌面板受限时请在浏览器中观看。",
|
||||
VIDEO_INVALID: "视频格式发生变化，请重新加载画面。",
|
||||
VIDEO_LIMIT: "观看面板数量已达上限，请关闭其他面板。",
|
||||
AUTH_FAILED: "访问密码错误，请重新输入。",
|
||||
AUTH_REQUIRED: "请输入访问密码，或在本机 setup 中配置凭据。",
|
||||
CONNECTION_FAILED: "连接失败，请检查设备是否在线及服务器设置。",
|
||||
CONNECTION_TIMEOUT: "等待远端画面超时，请检查远端状态后重试。",
|
||||
CONNECTION_LOST: "远程连接已中断。请检查远端状态并重新连接。",
|
||||
SESSION_LIMIT: "已有活动连接，请先断开当前电脑。",
|
||||
DEVICE_BUSY: "另一实例正在使用此设备。",
|
||||
CREDENTIAL_STORE_UNAVAILABLE: "系统凭据库不可用，密码未保存。",
|
||||
PAIRING_REQUIRED:
|
||||
"请在终端运行 Agent 工具结果中的 setup --open-panel 配对命令，再打开面板。",
|
||||
PANEL_UNAUTHORIZED: "面板授权已失效，请重新打开。",
|
||||
VIEW_UNAUTHORIZED: "此面板已失效，请使用最新打开的面板。",
|
||||
CONTROL_DENIED: "当前会话没有控制权限，或已被人工暂停。",
|
||||
NATIVE_PACKAGE_MISSING:
|
||||
"缺少对应平台的 CLI 运行组件，请检查包安装是否启用了 optional dependencies。",
|
||||
INVALID_ARGUMENT: "请检查设备 ID、服务器和输入内容。",
|
||||
DEVICE_ALIAS_CONFLICT: "这个备注已属于另一台设备，请换一个备注。",
|
||||
};
|
||||
function text(id: string, value: string) {
|
||||
if (element(id).textContent !== value) element(id).textContent = value;
|
||||
}
|
||||
function platformName(platform?: string) {
|
||||
return ({ macos: "macOS", windows: "Windows", linux: "Linux", android: "Android", ios: "iOS" } as Record<string, string>)[platform ?? ""] ?? "远程电脑";
|
||||
}
|
||||
function resetMetrics() {
|
||||
imageAt = 0;
|
||||
frameSize = frameRate = "";
|
||||
text("resolution", "—");
|
||||
text("fps", "—");
|
||||
element<HTMLButtonElement>("focus-view").disabled = true;
|
||||
}
|
||||
function setFocus(focused: boolean) {
|
||||
root.classList.toggle("focus-mode", focused);
|
||||
const button = element<HTMLButtonElement>("focus-view");
|
||||
const label = focused ? "还原布局" : "放大画面";
|
||||
button.textContent = label;
|
||||
if (focused) setDetails(false);
|
||||
button.setAttribute("aria-pressed", String(focused));
|
||||
button.setAttribute("aria-label", label);
|
||||
button.title = label;
|
||||
}
|
||||
function setDetails(open: boolean) {
|
||||
root.classList.toggle("details-open", open);
|
||||
element("details-toggle").setAttribute("aria-expanded", String(open));
|
||||
text("details-toggle", open ? "收起信息" : "连接信息");
|
||||
}
|
||||
element("details-toggle").onclick = () => setDetails(!root.classList.contains("details-open"));
|
||||
element("focus-view").onclick = () => setFocus(!root.classList.contains("focus-mode"));
|
||||
window.addEventListener("keydown", (event) => {
|
||||
if (event.key === "Escape" && root.classList.contains("focus-mode")) setFocus(false);
|
||||
});
|
||||
function hidePassword() {
|
||||
input("password").type = "password";
|
||||
text("password-toggle", "显示");
|
||||
element("password-toggle").setAttribute("aria-label", "显示密码");
|
||||
element("password-toggle").setAttribute("aria-pressed", "false");
|
||||
}
|
||||
element("password-toggle").onclick = () => {
|
||||
if (input("password").type === "text") return hidePassword();
|
||||
input("password").type = "text";
|
||||
text("password-toggle", "隐藏");
|
||||
element("password-toggle").setAttribute("aria-label", "隐藏密码");
|
||||
element("password-toggle").setAttribute("aria-pressed", "true");
|
||||
};
|
||||
select("mode").onchange = () => {
|
||||
text("mode-description", select("mode").value === "control"
|
||||
? "Agent 可以操作键鼠，你可以随时暂停。"
|
||||
: "Agent 可以查看屏幕，无法操作键鼠。");
|
||||
};
|
||||
function showError(error: unknown) {
|
||||
const code = error instanceof Error ? error.message : "INTERNAL_ERROR";
|
||||
element("error").textContent =
|
||||
messages[code] ??
|
||||
`操作未完成（${/^[A-Z_]+$/.test(code) ? code : "CONNECTION_ERROR"}）`;
|
||||
element("error").hidden = false;
|
||||
}
|
||||
function check(result: Data): Data {
|
||||
if (result.error) throw new Error(result.error.code ?? "INTERNAL_ERROR");
|
||||
return result;
|
||||
}
|
||||
async function devices() {
|
||||
const result = await invoke("devices");
|
||||
if (!started && result.defaults) {
|
||||
input("server").value = result.defaults.server;
|
||||
input("port").value = String(result.defaults.port);
|
||||
}
|
||||
const chosen = select("device").value || context?.device_alias || "";
|
||||
select("device").replaceChildren(new Option("连接新设备", ""));
|
||||
savedDevices = result.devices ?? [];
|
||||
for (const d of savedDevices)
|
||||
select("device").add(
|
||||
new Option(`${d.device_alias} · ${d.device_id}`, d.device_alias),
|
||||
);
|
||||
select("device").value = [...select("device").options].some(
|
||||
(o) => o.value === chosen,
|
||||
)
|
||||
? chosen
|
||||
: "";
|
||||
deviceChanged();
|
||||
}
|
||||
function deviceChanged() {
|
||||
const saved = !!select("device").value;
|
||||
element("new-device").hidden = saved;
|
||||
element("new-device-label").hidden = saved;
|
||||
input("server").disabled = input("port").disabled = saved;
|
||||
input("password").placeholder = saved
|
||||
? "留空使用已保存的密码"
|
||||
: "输入远端访问密码";
|
||||
}
|
||||
select("device").onchange = deviceChanged;
|
||||
input("save").onchange = () => {
|
||||
for (const name of ["remember", "authorize"]) {
|
||||
input(name).disabled = !input("save").checked;
|
||||
if (!input("save").checked) input(name).checked = false;
|
||||
}
|
||||
};
|
||||
function renderStatus(s: Data) {
|
||||
const connected = s.state === "ready" || s.state === "waiting_frame";
|
||||
const lost = ["closed", "failed"].includes(s.state);
|
||||
if (lost && !disconnectedByUser && ["ready", "waiting_frame"].includes(status.state)) showError(new Error("CONNECTION_LOST"));
|
||||
const changedSession = s.session_id !== status.session_id;
|
||||
status = s;
|
||||
const connecting = busy || s.state === "waiting_frame";
|
||||
root.dataset.connected = String(connected);
|
||||
root.dataset.connecting = String(connecting);
|
||||
root.dataset.controlling = String(connected && s.control === true && s.control_allowed === true && !s.human_paused);
|
||||
element("connection").dataset.state = connected ? "online" : connecting ? "connecting" : lost && !disconnectedByUser ? "offline" : "idle";
|
||||
text("connection-label", connected ? "已连接" : connecting ? "正在连接" : lost ? "未连接" : "等待连接");
|
||||
element("connect-form").hidden = connected;
|
||||
element("session-card").hidden = !connected;
|
||||
text("sidebar-title", connected ? "连接信息" : "连接设备");
|
||||
text("device-name", connected ? s.device_alias || "远程电脑" : "远程画面");
|
||||
if (!connected || changedSession) setDetails(false);
|
||||
text("permission", connected ? s.human_paused ? "控制已暂停" : s.control_allowed ? "允许 Agent 操作" : "仅查看画面" : "等待授权");
|
||||
element("permission").dataset.state = !connected ? "idle" : s.human_paused ? "paused" : "active";
|
||||
const platform = platformName(s.platform);
|
||||
text("session-info", connected ? `${platform} · 显示器 ${(s.display ?? 0) + 1}` : "等待设备连接");
|
||||
text("session-device", s.device_alias || "远程电脑");
|
||||
text("session-platform", platform);
|
||||
const device = savedDevices.find((d) => d.device_alias === s.device_alias);
|
||||
element("session-device-id-row").hidden = !device;
|
||||
text("session-device-id", device ? String(device.device_id).replace(/(\d{3})(?=\d)/g, "$1 ") : "");
|
||||
text("session-access", s.control_allowed ? "查看与操作" : "仅查看");
|
||||
text("session-display", `显示器 ${(s.display ?? 0) + 1}`);
|
||||
text("session-agent", s.human_paused ? "已暂停" : s.control_allowed ? "允许操作" : "仅可观察");
|
||||
element("session-agent").dataset.paused = String(!!s.human_paused);
|
||||
element<HTMLButtonElement>("pause").disabled = !connected || !s.control_allowed || controlBusy;
|
||||
const pauseLabel = s.human_paused ? "恢复控制" : "暂停控制";
|
||||
text("pause-label", pauseLabel);
|
||||
element<HTMLButtonElement>("disconnect").disabled = !s.session_id || controlBusy || busy;
|
||||
element<HTMLButtonElement>("connect").disabled = busy || connected || !invoke;
|
||||
element<HTMLFieldSetElement>("connection-fields").disabled = busy;
|
||||
element("connect").setAttribute("aria-busy", String(busy));
|
||||
text("connect-label", busy ? "正在连接…" : "连接");
|
||||
if (!connected) {
|
||||
element("frame").hidden = true;
|
||||
element("empty").hidden = false;
|
||||
element("stale").hidden = true;
|
||||
resetMetrics();
|
||||
if (root.classList.contains("focus-mode")) setFocus(false);
|
||||
} else if (changedSession) {
|
||||
resetMetrics();
|
||||
element("frame").hidden = true;
|
||||
element("empty").hidden = false;
|
||||
}
|
||||
if (!imageAt && !videoFailed) {
|
||||
text("empty-title", connecting || connected ? "正在建立远程连接" : lost ? "会话已结束" : "未连接设备");
|
||||
text("empty-description", connecting || connected ? "正在等待远端画面，请稍候。" : lost ? "请重新连接设备。" : "选择设备并输入访问密码。");
|
||||
text("video-state", connecting || connected ? "等待画面" : "待连接");
|
||||
element("video-state").dataset.state = "idle";
|
||||
}
|
||||
}
|
||||
async function connect() {
|
||||
if (!invoke || busy || ["ready", "waiting_frame"].includes(status.state)) return;
|
||||
element("error").hidden = true;
|
||||
disconnectedByUser = false;
|
||||
busy = true;
|
||||
renderStatus(status);
|
||||
const saved = select("device").value;
|
||||
let args: Data = {
|
||||
mode: select("mode").value,
|
||||
display_index: Number(input("display").value),
|
||||
save_device: input("save").checked,
|
||||
remember_password: input("remember").checked,
|
||||
persistent_authorization: input("authorize").checked,
|
||||
...(saved
|
||||
? { device_alias: saved }
|
||||
: {
|
||||
device_id: input("device-id").value.replace(/\s/g, ""),
|
||||
server: input("server").value,
|
||||
port: Number(input("port").value),
|
||||
...(input("label").value ? { label: input("label").value } : {}),
|
||||
}),
|
||||
};
|
||||
if (!saved || input("password").value)
|
||||
args.password = input("password").value;
|
||||
input("password").value = "";
|
||||
hidePassword();
|
||||
try {
|
||||
const result = await invoke("connect", args);
|
||||
args = {};
|
||||
renderStatus(result);
|
||||
if (result.persistence_error)
|
||||
showError(new Error(result.persistence_error));
|
||||
await devices();
|
||||
} catch (error) {
|
||||
args = {};
|
||||
showError(error);
|
||||
} finally {
|
||||
busy = false;
|
||||
renderStatus(status);
|
||||
}
|
||||
}
|
||||
element("connect").onclick = () => {
|
||||
void connect();
|
||||
};
|
||||
element("connect-form").addEventListener("keydown", (event) => {
|
||||
if (event.key === "Enter" && event.target instanceof HTMLInputElement) {
|
||||
event.preventDefault();
|
||||
void connect();
|
||||
}
|
||||
});
|
||||
element("pause").onclick = async () => {
|
||||
if (controlBusy) return;
|
||||
controlBusy = true;
|
||||
renderStatus(status);
|
||||
try {
|
||||
await invoke("control", {
|
||||
operation: status.human_paused ? "resume" : "pause",
|
||||
});
|
||||
renderStatus(await invoke("status"));
|
||||
} catch (e) {
|
||||
showError(e);
|
||||
} finally {
|
||||
controlBusy = false;
|
||||
renderStatus(status);
|
||||
}
|
||||
};
|
||||
element("disconnect").onclick = async () => {
|
||||
if (controlBusy) return;
|
||||
controlBusy = true;
|
||||
disconnectedByUser = true;
|
||||
renderStatus(status);
|
||||
try {
|
||||
await closePreview();
|
||||
await invoke("control", { operation: "disconnect" });
|
||||
element("error").hidden = true;
|
||||
resetMetrics();
|
||||
renderStatus(await invoke("status"));
|
||||
} catch (e) {
|
||||
disconnectedByUser = false;
|
||||
showError(e);
|
||||
} finally {
|
||||
controlBusy = false;
|
||||
renderStatus(status);
|
||||
}
|
||||
};
|
||||
async function closePreview() {
|
||||
++videoEpoch;
|
||||
player?.stop();
|
||||
player = undefined;
|
||||
videoSession = undefined;
|
||||
videoGeneration = undefined;
|
||||
videoStarting = videoFailed = false;
|
||||
element("video-retry").hidden = true;
|
||||
element("video-browser").hidden = true;
|
||||
}
|
||||
async function startPreview() {
|
||||
if (player || videoStarting || videoFailed || document.hidden) return;
|
||||
const epoch = ++videoEpoch;
|
||||
videoSession = status.session_id;
|
||||
videoGeneration = status.generation;
|
||||
videoStarting = true;
|
||||
const failed = (error: Error) => {
|
||||
if (epoch !== videoEpoch) return;
|
||||
videoFailed = true;
|
||||
showError(error);
|
||||
element("video-retry").hidden = false;
|
||||
element("video-browser").hidden = !CROSSDESK_APPS;
|
||||
element("stale").hidden = !imageAt;
|
||||
text("video-state", "画面不可用");
|
||||
element("video-state").dataset.state = "stale";
|
||||
if (!imageAt) {
|
||||
text("empty-title", "暂时无法显示画面");
|
||||
text("empty-description", "会话仍然保留。请尝试重新加载画面。");
|
||||
}
|
||||
};
|
||||
try {
|
||||
const grant = await invoke("video_open");
|
||||
if (epoch !== videoEpoch || document.hidden) return;
|
||||
player = new VideoPlayer(element<HTMLCanvasElement>("frame"), (info) => {
|
||||
if (epoch !== videoEpoch) return;
|
||||
imageAt = Date.now();
|
||||
element("frame").hidden = false;
|
||||
element("empty").hidden = true;
|
||||
element("stale").hidden = true;
|
||||
const size = `${info.width} × ${info.height}`;
|
||||
const rate = info.fps > 0 ? `${info.fps.toFixed(1)} FPS` : "测量中";
|
||||
if (size !== frameSize) { frameSize = size; text("resolution", size); }
|
||||
if (rate !== frameRate) { frameRate = rate; text("fps", rate); }
|
||||
element<HTMLButtonElement>("focus-view").disabled = false;
|
||||
text("video-state", "实时");
|
||||
element("video-state").dataset.state = "live";
|
||||
}, failed);
|
||||
player.start(grant.url, grant.ticket, grant.protocol, grant.decoder);
|
||||
} catch (error) {
|
||||
failed(error instanceof Error ? error : new Error("VIDEO_UNAVAILABLE"));
|
||||
} finally {
|
||||
if (epoch === videoEpoch) videoStarting = false;
|
||||
}
|
||||
}
|
||||
element("video-retry").onclick = async () => {
|
||||
await closePreview();
|
||||
element("error").hidden = true;
|
||||
await startPreview();
|
||||
};
|
||||
element("video-browser").onclick = async () => {
|
||||
try {
|
||||
const result = await app!.callServerTool({name: "crossdesk_open_browser", arguments: {view_id: context!.view_id}});
|
||||
if (result.isError) throw new Error("VIDEO_UNAVAILABLE");
|
||||
started = false;
|
||||
await closePreview();
|
||||
element("error").hidden = true;
|
||||
for (const name of ["connect", "pause", "disconnect"])
|
||||
element<HTMLButtonElement>(name).disabled = true;
|
||||
element("panel-status").hidden = false;
|
||||
element("panel-status").textContent = "浏览器观看面板已打开。";
|
||||
} catch (error) { showError(error); }
|
||||
};
|
||||
async function poll() {
|
||||
if (polling || !invoke || !started || document.hidden || busy || controlBusy) return;
|
||||
polling = true;
|
||||
try {
|
||||
renderStatus(await invoke("status"));
|
||||
if (videoSession && (videoSession !== status.session_id ||
|
||||
videoGeneration !== status.generation)) {
|
||||
if (videoFailed) element("error").hidden = true;
|
||||
await closePreview();
|
||||
}
|
||||
if (status.state !== "ready" || document.hidden) {
|
||||
await closePreview();
|
||||
return;
|
||||
}
|
||||
await startPreview();
|
||||
const stale = !!imageAt && Date.now() - imageAt > 2000;
|
||||
element("stale").hidden = !stale;
|
||||
if (stale) {
|
||||
text("video-state", "等待画面");
|
||||
element("video-state").dataset.state = "stale";
|
||||
text("fps", "—");
|
||||
frameRate = "";
|
||||
}
|
||||
} catch (error) { showError(error); }
|
||||
finally { polling = false; }
|
||||
}
|
||||
document.addEventListener("visibilitychange", () => {
|
||||
if (document.hidden) void closePreview();
|
||||
else void poll();
|
||||
});
|
||||
window.addEventListener("pagehide", () => {
|
||||
void closePreview();
|
||||
});
|
||||
setInterval(() => {
|
||||
void poll();
|
||||
}, 1000);
|
||||
async function ready(next: Data) {
|
||||
if (
|
||||
started &&
|
||||
context?.view_id === next.view_id &&
|
||||
context?.view_generation === next.view_generation
|
||||
)
|
||||
return;
|
||||
if (started) await closePreview();
|
||||
context = next;
|
||||
element("panel-status").hidden = true;
|
||||
started = false;
|
||||
await invoke("ready");
|
||||
await devices();
|
||||
started = true;
|
||||
element<HTMLButtonElement>("connect").disabled = false;
|
||||
await poll();
|
||||
}
|
||||
async function start() {
|
||||
if (CROSSDESK_APPS) {
|
||||
const { App } = await import("@modelcontextprotocol/ext-apps");
|
||||
app = new App({ name: "CrossDesk MCP", version: "0.1.0" }, {});
|
||||
const names: Record<string, string> = {
|
||||
ready: "crossdesk_ui_ready",
|
||||
devices: "crossdesk_ui_list_devices",
|
||||
status: "crossdesk_ui_get_session",
|
||||
connect: "crossdesk_ui_connect",
|
||||
video_open: "crossdesk_video_open",
|
||||
control: "crossdesk_viewer_control",
|
||||
};
|
||||
let connected = false,
|
||||
initial: Data | undefined;
|
||||
invoke = async (name, args = {}) => {
|
||||
const result = await app!.callServerTool({
|
||||
name: names[name]!,
|
||||
arguments: {
|
||||
...args,
|
||||
view_id: context!.view_id,
|
||||
view_generation: context!.view_generation,
|
||||
ui_token: context!.ui_token,
|
||||
},
|
||||
});
|
||||
const data =
|
||||
result.structuredContent ??
|
||||
JSON.parse((result.content?.[0] as Data)?.text ?? "{}");
|
||||
return check(data);
|
||||
};
|
||||
app.ontoolresult = (result) => {
|
||||
const meta = result._meta?.["crossdesk/ui"] as Data | undefined;
|
||||
if (!meta) return;
|
||||
initial = { ...(result.structuredContent ?? {}), ...meta };
|
||||
if (connected) void ready(initial).catch(showError);
|
||||
};
|
||||
app.onteardown = async () => {
|
||||
await closePreview();
|
||||
return {};
|
||||
};
|
||||
await app.connect();
|
||||
connected = true;
|
||||
if (initial) await ready(initial);
|
||||
} else {
|
||||
let csrf = "";
|
||||
const request = async (path: string, body?: Data) =>
|
||||
check(
|
||||
await (
|
||||
await fetch("/api/" + path, {
|
||||
method: body ? "POST" : "GET",
|
||||
credentials: "same-origin",
|
||||
headers: body
|
||||
? { "Content-Type": "application/json", "X-CrossDesk-CSRF": csrf }
|
||||
: {},
|
||||
body: body ? JSON.stringify(body) : undefined,
|
||||
cache: "no-store",
|
||||
})
|
||||
).json(),
|
||||
);
|
||||
const bootstrap = new URLSearchParams(location.hash.slice(1)).get(
|
||||
"bootstrap",
|
||||
);
|
||||
history.replaceState(null, "", location.pathname);
|
||||
const result = bootstrap
|
||||
? await request("bootstrap", { token: bootstrap })
|
||||
: await request("context");
|
||||
csrf = result.csrf;
|
||||
invoke = (name, args = {}) =>
|
||||
request(name, { ...args, view_id: context!.view_id });
|
||||
await ready(bootstrap ? result : result.views.at(-1));
|
||||
}
|
||||
}
|
||||
void start().catch(showError);
|
||||
Vendored
+1
-1
Submodule deps/submodules/minirtc updated: faa4adc1e3...9bdb5a775c
@@ -19,6 +19,11 @@
|
||||
#include <vector>
|
||||
|
||||
namespace crossdesk {
|
||||
struct EncodedVideoFrame {
|
||||
std::vector<uint8_t> data;
|
||||
uint64_t generation = 0, sequence = 0, timestamp_us = 0;
|
||||
bool key = false;
|
||||
};
|
||||
struct DecodedFrame {
|
||||
unsigned width = 0, height = 0;
|
||||
uint64_t generation = 0, frame_id = 0;
|
||||
@@ -46,6 +51,7 @@ struct SessionOptions {
|
||||
int port = 9099, display = 0, fps = 8;
|
||||
int preview_width = 1920, preview_height = 1080;
|
||||
int quality = 2, max_bitrate_bps = 6000000;
|
||||
bool hardware_acceleration = false;
|
||||
std::chrono::seconds timeout{30};
|
||||
};
|
||||
// All public methods except the destructor run on one owner thread. Call Tick
|
||||
@@ -59,6 +65,8 @@ public:
|
||||
void Tick();
|
||||
void Disconnect();
|
||||
bool SelectDisplay(int index);
|
||||
void SetVideoOutput(bool enabled, bool request_keyframe = false);
|
||||
std::optional<EncodedVideoFrame> TakeEncodedVideo();
|
||||
void SetPreview(int fps, int width, int height, int quality = -1,
|
||||
int max_bitrate_bps = -1);
|
||||
void SetControl(bool enabled);
|
||||
@@ -91,6 +99,8 @@ private:
|
||||
size_t, void *);
|
||||
static void Video(const MiniRtcVideoFrame *, const char *, size_t,
|
||||
const char *, size_t, void *);
|
||||
static int EncodedVideo(const uint8_t *, size_t, int64_t, const char *,
|
||||
size_t, const char *, size_t, void *);
|
||||
static void Network(const char *, size_t, TraversalMode,
|
||||
const MiniRtcNetTrafficStats *, const char *, size_t,
|
||||
void *);
|
||||
@@ -102,6 +112,7 @@ private:
|
||||
bool control_ = false, preview_supported_ = false, resolution_ = false;
|
||||
bool fps15_ = false, bandwidth_ = false, text_supported_ = false;
|
||||
bool preview_accepted_ = false;
|
||||
HostPlatform host_platform_ = HostPlatform::Unknown;
|
||||
std::vector<HostDisplay> displays_;
|
||||
std::set<int> keys_, buttons_;
|
||||
float pointer_x_ = .5f, pointer_y_ = .5f;
|
||||
@@ -114,6 +125,12 @@ private:
|
||||
MiniRtcNetTrafficStats network_{};
|
||||
TraversalMode traversal_ = UnknownMode;
|
||||
LatestFrame frames_;
|
||||
bool video_output_ = false, video_wait_key_ = true, video_request_key_ = false;
|
||||
size_t video_bytes_ = 0;
|
||||
uint64_t video_generation_ = 0, video_sequence_ = 0;
|
||||
std::deque<EncodedVideoFrame> video_frames_;
|
||||
std::vector<uint8_t> video_sps_, video_pps_;
|
||||
std::chrono::steady_clock::time_point video_key_requested_{};
|
||||
};
|
||||
nlohmann::json ParseBoundedJson(const std::string &text);
|
||||
std::string SafeText(const std::string &text, size_t limit = 160);
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
/*
|
||||
* @Author: DI JUNKUN
|
||||
* @Date: 2026-11-02
|
||||
* Copyright (c) 2026 by DI JUNKUN, All Rights Reserved.
|
||||
*/
|
||||
|
||||
#ifndef _FRAME_IMAGE_H_
|
||||
#define _FRAME_IMAGE_H_
|
||||
|
||||
#include <controller_session.h>
|
||||
|
||||
namespace crossdesk {
|
||||
struct ImageRegion {
|
||||
int x = 0, y = 0, width = 0, height = 0;
|
||||
};
|
||||
struct FrameImage {
|
||||
int width = 0, height = 0;
|
||||
ImageRegion region;
|
||||
std::vector<uint8_t> png;
|
||||
};
|
||||
// Pure bounded conversion; safe on an encoding worker, never calls the session.
|
||||
FrameImage EncodeFrameImage(const DecodedFrame &frame, ImageRegion region,
|
||||
int max_edge, size_t max_bytes);
|
||||
std::string ImageBase64(const std::vector<uint8_t> &bytes);
|
||||
} // namespace crossdesk
|
||||
|
||||
#endif
|
||||
@@ -77,7 +77,7 @@ Params ControllerSession::Parameters() {
|
||||
std::snprintf(p.log_path, sizeof(p.log_path), "%s",
|
||||
options_.log_directory.c_str());
|
||||
p.signal_server_port = options_.port;
|
||||
p.hardware_acceleration = false;
|
||||
p.hardware_acceleration = options_.hardware_acceleration;
|
||||
p.native_video_output = true;
|
||||
p.av1_encoding = false;
|
||||
p.turn_mode = TurnAutoUdpTcp;
|
||||
@@ -92,6 +92,7 @@ Params ControllerSession::Parameters() {
|
||||
p.on_net_status_report = Network;
|
||||
p.on_receive_data_buffer = Data;
|
||||
p.on_receive_video_frame = Video;
|
||||
p.on_receive_encoded_video = EncodedVideo;
|
||||
// No audio callback, clipboard receiver, file writer or incoming input
|
||||
// handler.
|
||||
p.user_id = "cli";
|
||||
@@ -118,6 +119,7 @@ void ControllerSession::Connect(SessionOptions options, std::string device,
|
||||
state_ = "registering";
|
||||
error_.clear();
|
||||
displays_.clear();
|
||||
host_platform_ = HostPlatform::Unknown;
|
||||
preview_supported_ = resolution_ = fps15_ = bandwidth_ = text_supported_ =
|
||||
false;
|
||||
preview_request_ = preview_ack_ = 0;
|
||||
@@ -238,6 +240,7 @@ void ControllerSession::Tick() {
|
||||
if (!a.from_json(e.data))
|
||||
continue;
|
||||
if (a.type == host_infomation) {
|
||||
host_platform_ = a.i.platform;
|
||||
displays_.clear();
|
||||
for (size_t i = 0; i < std::min(a.i.display_num, size_t(64)); ++i)
|
||||
displays_.push_back({SafeText(a.i.display_list[i]), a.i.left[i],
|
||||
@@ -342,6 +345,9 @@ void ControllerSession::Preview() {
|
||||
// 60 is the existing interactive-mode sentinel, not a 60 FPS request.
|
||||
// The encoder cadence is independently specified by frame_rate.
|
||||
s.frame_rate = 30;
|
||||
// Preserve desktop text detail at every cadence, including 30 FPS previews.
|
||||
s.preference =
|
||||
static_cast<int>(VideoDegradationPreference::MaintainResolution);
|
||||
s.audio = false;
|
||||
s.quality = options_.quality;
|
||||
if (resolution_) {
|
||||
@@ -419,7 +425,9 @@ std::optional<DecodedFrame> ControllerSession::TakeFrame(DecodedFrame *recycle)
|
||||
json ControllerSession::Status() const {
|
||||
json ds = json::array();
|
||||
for (size_t i = 0; i < displays_.size(); ++i)
|
||||
ds.push_back({{"index", i}, {"name", displays_[i].name}});
|
||||
ds.push_back({{"index", i}, {"name", displays_[i].name},
|
||||
{"left", displays_[i].left}, {"top", displays_[i].top},
|
||||
{"right", displays_[i].right}, {"bottom", displays_[i].bottom}});
|
||||
std::lock_guard lock(mutex_);
|
||||
return {{"state", state_},
|
||||
{"client_id", client_id_},
|
||||
@@ -427,6 +435,7 @@ json ControllerSession::Status() const {
|
||||
{"generation", generation_},
|
||||
{"control", control_},
|
||||
{"displays", ds},
|
||||
{"platform", HostPlatformName(host_platform_)},
|
||||
{"display", options_.display},
|
||||
{"text_input", text_supported_},
|
||||
{"preview_supported", preview_supported_},
|
||||
@@ -437,7 +446,19 @@ json ControllerSession::Status() const {
|
||||
{"preview_height", options_.preview_height},
|
||||
{"preview_quality", options_.quality},
|
||||
{"preview_bitrate_bps", options_.max_bitrate_bps},
|
||||
{"native_video_decoder", SafeText(std::string(
|
||||
network_.video_decoder,
|
||||
std::find(network_.video_decoder,
|
||||
network_.video_decoder + sizeof(network_.video_decoder), '\0')))},
|
||||
{"frames", frames_.Stats()},
|
||||
{"video_output",
|
||||
{{"enabled", video_output_},
|
||||
{"received", video_sequence_},
|
||||
{"queued", video_frames_.size()},
|
||||
{"queued_bytes", video_bytes_},
|
||||
{"waiting_keyframe", video_wait_key_},
|
||||
{"has_sps", !video_sps_.empty()},
|
||||
{"has_pps", !video_pps_.empty()}}},
|
||||
{"network",
|
||||
{{"video_bps", network_.video_inbound_stats.bitrate},
|
||||
{"rtt_ms", network_.rtt_ms},
|
||||
|
||||
@@ -0,0 +1,129 @@
|
||||
#include "controller_session.h"
|
||||
#include <algorithm>
|
||||
#include <display_stream_id.h>
|
||||
|
||||
namespace crossdesk {
|
||||
namespace {
|
||||
constexpr size_t kMaxAccessUnit = 4 * 1024 * 1024;
|
||||
// NAL spans include their Annex B start code, so cached parameter sets can be
|
||||
// prepended to an IDR without converting or re-encoding the original stream.
|
||||
size_t StartCode(const uint8_t *data, size_t size, size_t at) {
|
||||
if (at + 3 <= size && !data[at] && !data[at + 1]) {
|
||||
if (data[at + 2] == 1)
|
||||
return 3;
|
||||
if (at + 4 <= size && !data[at + 2] && data[at + 3] == 1)
|
||||
return 4;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
} // namespace
|
||||
void ControllerSession::SetVideoOutput(bool enabled, bool request_keyframe) {
|
||||
std::lock_guard lock(mutex_);
|
||||
if (video_output_ != enabled || request_keyframe) {
|
||||
video_frames_.clear();
|
||||
video_bytes_ = 0;
|
||||
video_wait_key_ = true;
|
||||
video_request_key_ = enabled;
|
||||
}
|
||||
video_output_ = enabled;
|
||||
}
|
||||
std::optional<EncodedVideoFrame> ControllerSession::TakeEncodedVideo() {
|
||||
std::lock_guard lock(mutex_);
|
||||
if (video_frames_.empty())
|
||||
return {};
|
||||
auto frame = std::move(video_frames_.front());
|
||||
video_frames_.pop_front();
|
||||
video_bytes_ -= frame.data.size();
|
||||
return frame;
|
||||
}
|
||||
int ControllerSession::EncodedVideo(const uint8_t *data, size_t size, int64_t,
|
||||
const char *, size_t, const char *stream,
|
||||
size_t stream_size, void *context) {
|
||||
auto &s = *static_cast<ControllerSession *>(context);
|
||||
try {
|
||||
std::lock_guard lock(s.mutex_);
|
||||
if (s.stopping_ || !s.video_output_ || !data || !size ||
|
||||
size > kMaxAccessUnit || !stream || stream_size >= 64 ||
|
||||
std::string(stream, stream_size) !=
|
||||
MakeDisplayStreamId(s.callback_display_))
|
||||
return 0;
|
||||
const auto now = std::chrono::steady_clock::now();
|
||||
if (s.video_generation_ != s.callback_generation_) {
|
||||
s.video_generation_ = s.callback_generation_;
|
||||
s.video_frames_.clear();
|
||||
s.video_bytes_ = 0;
|
||||
s.video_sps_.clear();
|
||||
s.video_pps_.clear();
|
||||
s.video_wait_key_ = s.video_request_key_ = true;
|
||||
}
|
||||
bool key = false;
|
||||
for (size_t at = 0; at + 3 < size;) {
|
||||
const auto prefix = StartCode(data, size, at);
|
||||
if (!prefix) {
|
||||
++at;
|
||||
continue;
|
||||
}
|
||||
const auto body = at + prefix;
|
||||
if (body >= size)
|
||||
break;
|
||||
size_t end = body + 1;
|
||||
while (end < size && !StartCode(data, size, end))
|
||||
++end;
|
||||
const auto type = data[body] & 31;
|
||||
key |= type == 5;
|
||||
if ((type == 7 || type == 8) && end - at <= 65536) {
|
||||
auto ¶meter = type == 7 ? s.video_sps_ : s.video_pps_;
|
||||
const std::vector<uint8_t> next(data + at, data + end);
|
||||
if (parameter != next) {
|
||||
parameter = next;
|
||||
s.video_wait_key_ = s.video_request_key_ = true;
|
||||
s.video_frames_.clear();
|
||||
s.video_bytes_ = 0;
|
||||
}
|
||||
}
|
||||
at = end;
|
||||
}
|
||||
++s.video_sequence_;
|
||||
const auto output_size =
|
||||
size + (key ? s.video_sps_.size() + s.video_pps_.size() : 0);
|
||||
if (s.video_frames_.size() >= 60 ||
|
||||
s.video_bytes_ + output_size > 8 * 1024 * 1024) {
|
||||
s.video_frames_.clear();
|
||||
s.video_bytes_ = 0;
|
||||
s.video_wait_key_ = s.video_request_key_ = true;
|
||||
}
|
||||
if (key && !s.video_sps_.empty() && !s.video_pps_.empty()) {
|
||||
s.video_wait_key_ = s.video_request_key_ = false;
|
||||
}
|
||||
if (!s.video_wait_key_) {
|
||||
EncodedVideoFrame frame;
|
||||
frame.key = key;
|
||||
frame.generation = s.callback_generation_;
|
||||
frame.sequence = s.video_sequence_;
|
||||
frame.timestamp_us =
|
||||
std::chrono::duration_cast<std::chrono::microseconds>(
|
||||
now.time_since_epoch())
|
||||
.count();
|
||||
if (key) {
|
||||
frame.data = s.video_sps_;
|
||||
frame.data.insert(frame.data.end(), s.video_pps_.begin(),
|
||||
s.video_pps_.end());
|
||||
}
|
||||
frame.data.insert(frame.data.end(), data, data + size);
|
||||
if (frame.data.size() <= kMaxAccessUnit) {
|
||||
s.video_bytes_ += frame.data.size();
|
||||
s.video_frames_.push_back(std::move(frame));
|
||||
} else {
|
||||
s.video_wait_key_ = s.video_request_key_ = true;
|
||||
}
|
||||
}
|
||||
if (s.video_request_key_ &&
|
||||
now - s.video_key_requested_ >= std::chrono::seconds(1)) {
|
||||
s.video_key_requested_ = now;
|
||||
return 1;
|
||||
}
|
||||
} catch (...) { /* The receive callback must not unwind into MiniRTC. */
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
} // namespace crossdesk
|
||||
@@ -0,0 +1,85 @@
|
||||
#include "frame_image.h"
|
||||
#include <algorithm>
|
||||
#include <libyuv.h>
|
||||
#include <png.h>
|
||||
#include <stdexcept>
|
||||
|
||||
namespace crossdesk {
|
||||
FrameImage EncodeFrameImage(const DecodedFrame &f, ImageRegion r, int max_edge,
|
||||
size_t max_bytes) {
|
||||
if (!f.width || !f.height || f.width > 8192 || f.height > 8192 ||
|
||||
(f.width & 1) || (f.height & 1) ||
|
||||
uint64_t(f.width) * f.height > 16777216 ||
|
||||
f.nv12.size() != size_t(f.width) * f.height * 3 / 2 || max_edge < 64 ||
|
||||
max_edge > 2048 || max_bytes < 4096 || max_bytes > 2 * 1024 * 1024)
|
||||
throw std::invalid_argument("INVALID_IMAGE_REQUEST");
|
||||
if (!r.width && !r.height)
|
||||
r = {0, 0, int(f.width), int(f.height)};
|
||||
if (r.x < 0 || r.y < 0 || r.width < 1 || r.height < 1 ||
|
||||
r.width > int(f.width) || r.height > int(f.height) ||
|
||||
r.x > int(f.width) - r.width || r.y > int(f.height) - r.height)
|
||||
throw std::invalid_argument("INVALID_REGION");
|
||||
std::vector<uint8_t> argb(size_t(f.width) * f.height * 4);
|
||||
if (libyuv::NV12ToARGB(f.nv12.data(), f.width,
|
||||
f.nv12.data() + size_t(f.width) * f.height, f.width,
|
||||
argb.data(), f.width * 4, f.width, f.height))
|
||||
throw std::runtime_error("IMAGE_CONVERSION_FAILED");
|
||||
FrameImage result;
|
||||
result.region = r;
|
||||
for (int edge = max_edge;; edge = std::max(32, edge * 3 / 4)) {
|
||||
const double scale =
|
||||
std::min(1.0, double(edge) / std::max(r.width, r.height));
|
||||
result.width = std::max(1, int(r.width * scale));
|
||||
result.height = std::max(1, int(r.height * scale));
|
||||
std::vector<uint8_t> scaled(size_t(result.width) * result.height * 4);
|
||||
std::vector<uint8_t> rgb(size_t(result.width) * result.height * 3);
|
||||
if (libyuv::ARGBScale(argb.data() + (size_t(r.y) * f.width + r.x) * 4,
|
||||
f.width * 4, r.width, r.height, scaled.data(),
|
||||
result.width * 4, result.width, result.height,
|
||||
libyuv::kFilterBox) ||
|
||||
libyuv::ARGBToRAW(scaled.data(), result.width * 4, rgb.data(),
|
||||
result.width * 3, result.width, result.height))
|
||||
throw std::runtime_error("IMAGE_CONVERSION_FAILED");
|
||||
png_image image{};
|
||||
image.version = PNG_IMAGE_VERSION;
|
||||
image.width = result.width;
|
||||
image.height = result.height;
|
||||
image.format = PNG_FORMAT_RGB;
|
||||
png_alloc_size_t size = 0;
|
||||
if (!png_image_write_to_memory(&image, nullptr, &size, 0, rgb.data(), 0,
|
||||
nullptr)) {
|
||||
png_image_free(&image);
|
||||
throw std::runtime_error("IMAGE_ENCODING_FAILED");
|
||||
}
|
||||
if (size <= max_bytes) {
|
||||
result.png.resize(size);
|
||||
const auto ok = png_image_write_to_memory(
|
||||
&image, result.png.data(), &size, 0, rgb.data(), 0, nullptr);
|
||||
png_image_free(&image);
|
||||
if (!ok)
|
||||
throw std::runtime_error("IMAGE_ENCODING_FAILED");
|
||||
result.png.resize(size);
|
||||
return result;
|
||||
}
|
||||
png_image_free(&image);
|
||||
if (edge == 32)
|
||||
throw std::runtime_error("IMAGE_TOO_LARGE");
|
||||
}
|
||||
}
|
||||
std::string ImageBase64(const std::vector<uint8_t> &data) {
|
||||
constexpr char alphabet[] =
|
||||
"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
|
||||
std::string out;
|
||||
out.reserve((data.size() + 2) / 3 * 4);
|
||||
for (size_t i = 0; i < data.size(); i += 3) {
|
||||
const uint32_t n = (uint32_t(data[i]) << 16) |
|
||||
(i + 1 < data.size() ? uint32_t(data[i + 1]) << 8 : 0) |
|
||||
(i + 2 < data.size() ? data[i + 2] : 0);
|
||||
out += alphabet[n >> 18];
|
||||
out += alphabet[(n >> 12) & 63];
|
||||
out += i + 1 < data.size() ? alphabet[(n >> 6) & 63] : '=';
|
||||
out += i + 2 < data.size() ? alphabet[n & 63] : '=';
|
||||
}
|
||||
return out;
|
||||
}
|
||||
} // namespace crossdesk
|
||||
@@ -58,6 +58,8 @@ struct Grid {
|
||||
Rect Fit(int width, int height, int columns, int pixel_rows,
|
||||
double pixel_aspect = 1);
|
||||
Grid Layout(const DecodedFrame &, Viewport);
|
||||
// Calculate crop and pointer geometry without decoded pixel storage.
|
||||
Grid Layout(unsigned source_width, unsigned source_height, Viewport);
|
||||
Grid Convert(const DecodedFrame &, Viewport);
|
||||
std::optional<std::pair<float, float>> MapPointer(const Grid &, int column,
|
||||
int row);
|
||||
@@ -119,9 +121,11 @@ private:
|
||||
bool pasting_ = false;
|
||||
unsigned mouse_buttons_ = 0;
|
||||
float pointer_x_ = .5f, pointer_y_ = .5f;
|
||||
int mouse_column_ = -1, mouse_row_ = -1;
|
||||
void Tap(ControllerSession &, int key, int modifier = 0);
|
||||
bool LocalKey(int key, ControllerSession &, const Grid *);
|
||||
void Navigate(char key, const Grid *);
|
||||
void Zoom(double next_zoom, const Grid *);
|
||||
void Mouse(int button, int x, int y, bool release, ControllerSession &,
|
||||
const Grid *);
|
||||
};
|
||||
|
||||
+58
-14
@@ -45,8 +45,7 @@ bool Input::LocalKey(int key, ControllerSession &s, const Grid *grid) {
|
||||
} else if (key == 'c') {
|
||||
control = !control;
|
||||
} else if (key == '+' || key == '=' || key == '-' || key == 'f' ||
|
||||
key == 'd' || key == 'i' || key == 'j' || key == 'k' ||
|
||||
key == 'l') {
|
||||
key == 'i' || key == 'j' || key == 'k' || key == 'l') {
|
||||
Navigate(char(key), grid);
|
||||
} else if (!command_mode) {
|
||||
return false;
|
||||
@@ -58,20 +57,52 @@ bool Input::LocalKey(int key, ControllerSession &s, const Grid *grid) {
|
||||
mouse_buttons_ = 0;
|
||||
return true;
|
||||
}
|
||||
void Input::Zoom(double next_zoom, const Grid *grid) {
|
||||
if (next_zoom == zoom)
|
||||
return;
|
||||
if (grid && mouse_column_ >= 0 && mouse_row_ >= 0) {
|
||||
auto v = grid->viewport;
|
||||
v.zoom = zoom;
|
||||
v.native = native;
|
||||
v.center_x = center_x;
|
||||
v.center_y = center_y;
|
||||
// Rebuild from the requested view so queued key/wheel repeats compose
|
||||
// even when the terminal is still displaying an earlier frame.
|
||||
const auto before = Layout(grid->source_width, grid->source_height, v);
|
||||
if (const auto anchor = MapPointer(before, mouse_column_, mouse_row_)) {
|
||||
v.zoom = next_zoom;
|
||||
const auto after = Layout(grid->source_width, grid->source_height, v);
|
||||
if (const auto moved = MapPointer(after, mouse_column_, mouse_row_)) {
|
||||
const auto center = [](int origin, int size, unsigned extent,
|
||||
double shift) {
|
||||
const double half = size * .5 / extent;
|
||||
return std::clamp((origin + size * .5 + shift * (extent - 1)) /
|
||||
extent,
|
||||
half, 1 - half);
|
||||
};
|
||||
// Use the actual, clamped crop center and the same source-pixel
|
||||
// mapping as remote input. Letterboxing and status rows stay aligned.
|
||||
center_x = center(after.source.x, after.source.width,
|
||||
after.source_width,
|
||||
double(anchor->first) - moved->first);
|
||||
center_y = center(after.source.y, after.source.height,
|
||||
after.source_height,
|
||||
double(anchor->second) - moved->second);
|
||||
}
|
||||
}
|
||||
}
|
||||
zoom = next_zoom;
|
||||
}
|
||||
void Input::Navigate(char key, const Grid *grid) {
|
||||
if (key == '+' || key == '=')
|
||||
zoom = std::min(32.0, zoom * 2);
|
||||
Zoom(std::min(32.0, zoom * 2), grid);
|
||||
if (key == '-')
|
||||
zoom = std::max(1.0, zoom / 2);
|
||||
Zoom(std::max(1.0, zoom / 2), grid);
|
||||
if (key == 'f') {
|
||||
native = false;
|
||||
zoom = 1;
|
||||
center_x = center_y = .5;
|
||||
}
|
||||
if (key == 'd') {
|
||||
native = true;
|
||||
zoom = 1;
|
||||
}
|
||||
// Use the displayed crop as the pan step, including native pixel mode.
|
||||
const double dx = grid && grid->source_width
|
||||
? double(grid->source.width) / grid->source_width
|
||||
@@ -99,14 +130,26 @@ void Input::Mouse(int b, int x, int y, bool release, ControllerSession &s,
|
||||
const Grid *grid) {
|
||||
++mouse_reports;
|
||||
// Extra mouse buttons must not alias the three supported remote buttons.
|
||||
if (!grid || command_mode || b < 0 || b >= 128)
|
||||
if (b < 0 || b >= 128 || x <= 0 || y <= 0)
|
||||
return;
|
||||
auto pos = MapPointer(*grid, x - 1, y - 1);
|
||||
// Track terminal coordinates even in local command mode. Re-map them for
|
||||
// every zoom; a stationary mouse must follow subsequent view changes.
|
||||
mouse_column_ = x - 1;
|
||||
mouse_row_ = y - 1;
|
||||
if (!grid || command_mode)
|
||||
return;
|
||||
auto pos = MapPointer(*grid, mouse_column_, mouse_row_);
|
||||
if (!control) {
|
||||
if (pos && !release && ((b & 64) || b == 0)) {
|
||||
center_x = pos->first;
|
||||
center_y = pos->second;
|
||||
Navigate((b & 64) ? ((b & 1) ? '-' : '+') : 'd', grid);
|
||||
if (pos && !release) {
|
||||
if (b & 64) {
|
||||
Navigate((b & 1) ? '-' : '+', grid);
|
||||
} else if (b == 0) {
|
||||
center_x = pos->first;
|
||||
center_y = pos->second;
|
||||
native = true;
|
||||
zoom = 1;
|
||||
++view_revision;
|
||||
}
|
||||
}
|
||||
return;
|
||||
}
|
||||
@@ -248,6 +291,7 @@ void Input::Feed(const std::string &bytes, ControllerSession &s,
|
||||
control = false;
|
||||
s.SetControl(false);
|
||||
mouse_buttons_ = 0;
|
||||
mouse_column_ = mouse_row_ = -1;
|
||||
continue;
|
||||
}
|
||||
int key = 0;
|
||||
|
||||
@@ -25,21 +25,25 @@ static uint32_t Pixel(const DecodedFrame &f, int x, int y) {
|
||||
(clamp((298 * yy - 100 * u - 208 * v + 128) >> 8) << 8) |
|
||||
clamp((298 * yy + 516 * u + 128) >> 8);
|
||||
}
|
||||
static void Validate(const DecodedFrame &f, const Viewport &v) {
|
||||
static void Validate(unsigned width, unsigned height, const Viewport &v) {
|
||||
if (v.columns < 1 || v.rows < 1 || v.columns > 240 || v.rows > 100 ||
|
||||
v.row_offset < 0 || v.row_offset > 100 ||
|
||||
!std::isfinite(v.cell_aspect) || v.cell_aspect < .2 ||
|
||||
v.cell_aspect > 2 || !std::isfinite(v.zoom) || v.zoom < 1 ||
|
||||
v.zoom > 32 || !std::isfinite(v.center_x) || !std::isfinite(v.center_y) ||
|
||||
v.center_x < 0 || v.center_x > 1 || v.center_y < 0 || v.center_y > 1 ||
|
||||
!f.width || !f.height || f.width > 8192 || f.height > 8192 ||
|
||||
(f.width & 1) || (f.height & 1) ||
|
||||
size_t(f.width) * f.height > 16777216 ||
|
||||
f.nv12.size() < size_t(f.width) * f.height * 3 / 2)
|
||||
!width || !height || width > 8192 || height > 8192 || (width & 1) ||
|
||||
(height & 1) || size_t(width) * height > 16777216)
|
||||
throw std::invalid_argument("invalid frame or viewport");
|
||||
}
|
||||
static Rect Crop(const DecodedFrame &f, const Viewport &v) {
|
||||
double width = f.width, height = f.height;
|
||||
static void Validate(const DecodedFrame &f, const Viewport &v) {
|
||||
Validate(f.width, f.height, v);
|
||||
if (f.nv12.size() < size_t(f.width) * f.height * 3 / 2)
|
||||
throw std::invalid_argument("invalid frame or viewport");
|
||||
}
|
||||
static Rect Crop(unsigned source_width, unsigned source_height,
|
||||
const Viewport &v) {
|
||||
double width = source_width, height = source_height;
|
||||
if (v.native) {
|
||||
const int sx = v.utf8 && v.detail == AnsiDetail::Quadrant ? 2 : 1;
|
||||
const int sy = v.utf8 ? 2 : 1;
|
||||
@@ -60,8 +64,10 @@ static Rect Crop(const DecodedFrame &f, const Viewport &v) {
|
||||
}
|
||||
int w = std::max(1, int(width / v.zoom)),
|
||||
h = std::max(1, int(height / v.zoom));
|
||||
int x = std::clamp(int(v.center_x * f.width - w * .5), 0, int(f.width) - w);
|
||||
int y = std::clamp(int(v.center_y * f.height - h * .5), 0, int(f.height) - h);
|
||||
int x = std::clamp(int(v.center_x * source_width - w * .5), 0,
|
||||
int(source_width) - w);
|
||||
int y = std::clamp(int(v.center_y * source_height - h * .5), 0,
|
||||
int(source_height) - h);
|
||||
return {x, y, w, h};
|
||||
}
|
||||
// Area resampling integrates the entire source footprint. Thin lines cannot
|
||||
@@ -249,16 +255,20 @@ static void AppendNumber(std::string &out, uint32_t value) {
|
||||
}
|
||||
Grid Layout(const DecodedFrame &f, Viewport v) {
|
||||
Validate(f, v);
|
||||
return Layout(f.width, f.height, v);
|
||||
}
|
||||
Grid Layout(unsigned source_width, unsigned source_height, Viewport v) {
|
||||
Validate(source_width, source_height, v);
|
||||
const int sx = v.utf8 && v.detail == AnsiDetail::Quadrant ? 2 : 1,
|
||||
sy = v.utf8 ? 2 : 1;
|
||||
const int width = v.columns * sx, height = v.rows * sy;
|
||||
Rect crop = Crop(f, v);
|
||||
Rect crop = Crop(source_width, source_height, v);
|
||||
Grid g{v,
|
||||
Fit(crop.width, crop.height, width, height, v.cell_aspect * sy / sx),
|
||||
{}};
|
||||
g.source = crop;
|
||||
g.source_width = f.width;
|
||||
g.source_height = f.height;
|
||||
g.source_width = source_width;
|
||||
g.source_height = source_height;
|
||||
g.samples_x = sx;
|
||||
g.samples_y = sy;
|
||||
return g;
|
||||
@@ -553,7 +563,7 @@ PixelFrame ImagePixels(const DecodedFrame &f, const Viewport &requested) {
|
||||
pixels.height =
|
||||
std::clamp(v.pixel_height > 0 ? v.pixel_height : v.rows * 20, 2, 1080);
|
||||
pixels.rgb.resize(size_t(pixels.width) * pixels.height * 3, 0);
|
||||
const auto crop = Crop(f, v);
|
||||
const auto crop = Crop(f.width, f.height, v);
|
||||
const auto r = Fit(crop.width, crop.height, pixels.width, pixels.height,
|
||||
double(v.columns) * v.cell_aspect * pixels.height /
|
||||
(v.rows * pixels.width));
|
||||
@@ -678,7 +688,7 @@ std::string Kitty(const DecodedFrame &f, const Viewport &requested,
|
||||
std::clamp(v.pixel_width > 0 ? v.pixel_width : v.columns * 10, 2, 1920);
|
||||
int h =
|
||||
std::clamp(v.pixel_height > 0 ? v.pixel_height : v.rows * 20, 2, 1080);
|
||||
const auto crop = Crop(f, v);
|
||||
const auto crop = Crop(f.width, f.height, v);
|
||||
std::string b64;
|
||||
for (;;) {
|
||||
auto r = Fit(crop.width, crop.height, w, h,
|
||||
|
||||
@@ -79,41 +79,52 @@ std::vector<std::string> StatusLines(const StatusBar &bar, int columns) {
|
||||
|
||||
std::string keys, view;
|
||||
if (inner >= 63) {
|
||||
const size_t count = inner >= 99 ? 4 : inner >= 77 ? 3 : 2;
|
||||
const size_t count = inner >= 99 ? 4 : inner >= 89 ? 3 : 2;
|
||||
if (bar.command_mode) {
|
||||
keys =
|
||||
Columns("LOCAL",
|
||||
count >= 3 ? std::vector<std::string>{"<Esc/Ctrl+C> Return",
|
||||
"<c> Toggle control",
|
||||
"<q> Quit"}
|
||||
: std::vector<std::string>{"<Esc/Ctrl+C> Return",
|
||||
"<c> Toggle; <q> Quit"},
|
||||
inner, count);
|
||||
keys = Columns(
|
||||
"LOCAL",
|
||||
count == 4
|
||||
? std::vector<std::string>{"<Esc/Ctrl+C> Return",
|
||||
"<c> Toggle control", "<q> Quit",
|
||||
"<mouse> Set anchor"}
|
||||
: count == 3
|
||||
? std::vector<std::string>{"<Esc/Ctrl+C> Return",
|
||||
"<c> Toggle control", "<q> Quit"}
|
||||
: std::vector<std::string>{"<Esc/Ctrl+C> Return",
|
||||
"<c> Toggle; <q> Quit"},
|
||||
inner, count);
|
||||
} else if (bar.control) {
|
||||
keys = Columns("KEYS", {"<Ctrl+]> Commands", "Input -> remote"}, inner,
|
||||
count);
|
||||
} else {
|
||||
keys = Columns("KEYS",
|
||||
count >= 3
|
||||
? std::vector<std::string>{"<c> Control", "<q> Quit",
|
||||
"<Ctrl+]> Commands"}
|
||||
: std::vector<std::string>{"<c> Control; <q> Quit",
|
||||
"<Ctrl+]> Commands"},
|
||||
inner, count);
|
||||
keys = Columns(
|
||||
"KEYS",
|
||||
count == 4
|
||||
? std::vector<std::string>{"<c> Control", "<q> Quit",
|
||||
"<Ctrl+]> Commands", "<click> Detail"}
|
||||
: count == 3
|
||||
? std::vector<std::string>{"<c> Control; <q> Quit",
|
||||
"<Ctrl+]> Commands", "<click> Detail"}
|
||||
: std::vector<std::string>{"<c> Control; <q> Quit",
|
||||
"<Ctrl+]> Commands"},
|
||||
inner, count);
|
||||
}
|
||||
if (bar.control && !bar.command_mode) {
|
||||
view = Columns("THEN", {"<c> Toggle control", "<q> Quit"}, inner, count);
|
||||
} else {
|
||||
const auto zoom = bar.command_mode ? "<+/-> Zoom at mouse"
|
||||
: "<+/wheel/-> Zoom at mouse";
|
||||
view = Columns(
|
||||
"VIEW",
|
||||
count == 4
|
||||
? std::vector<std::string>{"<arrows/ijkl> Pan", "<+/-> Zoom",
|
||||
"<f> Fit", "<d> Detail"}
|
||||
? std::vector<std::string>{"<+/-> Zoom at mouse",
|
||||
bar.command_mode ? "<=> Zoom in"
|
||||
: "<wheel> Zoom",
|
||||
"<arrows/ijkl> Pan", "<f> Full view"}
|
||||
: count == 3
|
||||
? std::vector<std::string>{"<arrows/ijkl> Pan", "<+/-> Zoom",
|
||||
"<f> Fit; <d> Detail"}
|
||||
: std::vector<std::string>{"<arrows> Pan; <+/-> Zoom",
|
||||
"<f> Fit; <d> Detail"},
|
||||
? std::vector<std::string>{zoom, "<arrows/ijkl> Pan",
|
||||
"<f> Full view"}
|
||||
: std::vector<std::string>{zoom, "<arrows> Pan; <f> Full"},
|
||||
inner, count);
|
||||
}
|
||||
} else {
|
||||
@@ -139,11 +150,18 @@ std::vector<std::string> StatusLines(const StatusBar &bar, int columns) {
|
||||
? std::vector<std::string>{"<c> Toggle control | <q> Quit",
|
||||
"<c> Toggle; <q> Quit", "<c> / <q>",
|
||||
"<q>", "q"}
|
||||
: std::vector<std::string>{"<arrows> Pan | <+/-> Zoom | <f> Fit | "
|
||||
"<d> Detail",
|
||||
"<+/-> Zoom | <f> Fit | <d> Detail",
|
||||
"<f> Fit | <d> Detail", "<f> Fit", "<f>",
|
||||
"f"},
|
||||
: bar.command_mode
|
||||
? std::vector<
|
||||
std::string>{"<+/-> Zoom at mouse | <arrows> Pan | <f> Full",
|
||||
"<+/-> Zoom at mouse | <f> Full",
|
||||
"<+/-> Zoom | <f> Full", "<+/-> | <f>", "<+/->",
|
||||
"+"}
|
||||
: std::vector<std::string>{"<+/wheel/-> Zoom at mouse | <arrows> "
|
||||
"Pan | <f> Full",
|
||||
"<+/wheel/-> Zoom at mouse | <f> Full",
|
||||
"<+/wheel/-> Zoom | <f> Full",
|
||||
"<+/-> Zoom | <f> Full", "<+/-> | <f>",
|
||||
"<+/->", "+"},
|
||||
inner);
|
||||
}
|
||||
std::vector<std::string> lines{status, keys, view};
|
||||
|
||||
Reference in New Issue
Block a user