diff --git a/apps/cli/docs/bridge-protocol.md b/apps/cli/docs/bridge-protocol.md index 4515ebcc..d73ddb37 100644 --- a/apps/cli/docs/bridge-protocol.md +++ b/apps/cli/docs/bridge-protocol.md @@ -8,7 +8,7 @@ | method | 字段 | 行为 | |---|---|---| | hello | version:1 | 协议版本握手 | -| viewport | columns:1..240, rows:1..100, colors:8/24/256, utf8:boolean, fps:1..30；可选 detail:quadrant/half, quality:low/balanced/high/ultra, zoom:1..32, native:boolean | 设置画面行数（不含状态栏）、本地目标与预览档位；默认 quadrant/high/1、native=false，保留未显式指定的 detail/quality/zoom/native；不探测宿主 TTY | +| viewport | columns:1..240, rows:1..100, colors:8/24/256, utf8:boolean, fps:1..30；可选 cell_aspect:0.2..2, detail:quadrant/half, quality:low/balanced/high/ultra, zoom:1..32, native:boolean | 设置画面行数（不含状态栏）、字符宽高比、本地目标与预览档位；默认 cell_aspect=0.5、quadrant/high/1、native=false，保留未显式指定的 cell_aspect/detail/quality/zoom/native；不探测宿主 TTY | | connect | device:9位数字字符串, display:0..63（默认0） | 从专用 password fd 读取至换行/EOF（最多256字节、等待5秒），开始异步注册/连接；ok不是远端已连接 | | status | 无 | 返回当前状态 | | control | enabled:boolean | 显式启停控制；关闭时释放输入 | @@ -23,10 +23,12 @@ 帧事件： ```json -{"event":"frame","frame_id":42,"decoded_frame_id":38,"generation":4,"source_width":1280,"source_height":720,"zoom":2,"native":false,"source_rect":[320,180,640,360],"detail":"quadrant","columns":80,"rows":23,"lines":["..."],"queue_ms":3.2} +{"event":"frame","frame_id":42,"decoded_frame_id":38,"generation":4,"source_width":1280,"source_height":720,"zoom":2,"native":false,"source_rect":[320,180,640,360],"detail":"quadrant","columns":80,"rows":23,"cell_aspect":0.45,"lines":["..."],"queue_ms":3.2} ``` -实际 lines 数为 rows，仅包含本地生成的 SGR 颜色序列、空格和 U+2580～U+259F 块字符，不含远端名称或任意终端指令。`frame_id` 是进程递增的输出序号，静止画面重绘仍产生新序号；`decoded_frame_id` 标识源解码帧，同源重绘可以相同。`source_width/height` 为实际解码尺寸，source_rect 为实际源裁剪矩形 [x,y,width,height]。native=true 表示原始细节模式，zoom 为当前模式下的倍率；generation 在关闭、重新连接和显示器切换时变化。viewport 变化可能与已写出的旧尺寸帧交错，宿主应丢弃不匹配尺寸/旧 generation 的帧并释放对应 ACK。宿主返回的 ACK 表示消费/布局，不表示屏幕物理显示完成。 +实际 lines 数为 rows，仅包含本地生成的 SGR 颜色序列、空格和 U+2580～U+259F 块字符，不含远端名称或任意终端指令。`frame_id` 是进程递增的输出序号，静止画面重绘仍产生新序号；`decoded_frame_id` 标识源解码帧，同源重绘可以相同。`source_width/height` 为实际解码尺寸，source_rect 为实际源裁剪矩形 [x,y,width,height]。native=true 表示原始细节模式，zoom 为当前模式下的倍率；generation 在关闭、重新连接和显示器切换时变化。viewport 变化可能与已写出的旧尺寸帧交错，宿主应丢弃不匹配行列数／cell_aspect 或旧 generation 的帧并释放对应 ACK。宿主返回的 ACK 表示消费/布局，不表示屏幕物理显示完成。 + +`cell_aspect` 为单个字符的宽／高，必须是 0.2～2 范围内的有限 JSON 数值；例如 9×20 像素字符应传 0.45。宿主负责测量或提供手动设置，bridge 根据该比例计算图像布局与鼠标映射，响应及状态事件中的 `status.cell_aspect` 返回当前值。该字段为 v1 的可选扩展；省略时保留上次值，首次默认 0.5。旧版 bridge 的帧可能不含此字段。修改比例也会触发静止画面重绘，新的鼠标映射仍在帧 ACK 后生效。 状态事件每秒最多一次：`{"event":"state","status":{...},"output_bytes":123,"submitted_frames":3,"elapsed_seconds":1.5}`。status 含 registering/connecting/connected/failed/closed/idle、client_id（当前或刚结束会话的 `cli-xxxxxxxxx`，注册前为空）、error、control、generation、显示器、文本/预览能力、预览确认状态、请求的 preview_width/height/quality/bitrate_bps、帧计数与网络统计。接收/提交 FPS 由相应计数的增量除以时间增量计算。 diff --git a/apps/cli/docs/terminal-client.md b/apps/cli/docs/terminal-client.md index 1150f36e..c2538d35 100644 --- a/apps/cli/docs/terminal-client.md +++ b/apps/cli/docs/terminal-client.md @@ -45,7 +45,13 @@ crossdesk-cli 123456789 --server your-server.example --port 9099 ANSI 默认使用四象限字符：每个字符从 2×2 样本选择误差最小的两个颜色和象限形状，比原先上下半方块多一倍水平采样点；每个字符仍只有前景/背景两色。使用 `--ansi-detail half` 可回到上下半方块以兼容字体。缩小时采用面积加权滤波，避免最近邻采样漏掉细线。支持 TrueColor、256 色和基础 8 色，256 色选择实际颜色立方体或灰阶条中更接近的颜色。环境的 COLORTERM/TERM 决定颜色模式；非 UTF-8 locale 使用背景色空格。TERM=dumb、非 TTY 输出明确拒绝交互连接，不输出全屏转义序列。 -画面按字符单元比例等比缩放，留黑边及状态栏，默认假定字符宽高比 0.5；TTY 提供像素尺寸时使用其比例。每次只输出变化行，合并连续颜色设置；缩放在完整输出边界清理旧画面。上限 240 列、100 画面行；默认顶部预留一行状态栏、底部预留两行快捷键，另预留一列避免自动换行。顶部状态栏用反色区分，左侧为控制模式和连接进度，右侧为画面参数及网络信息；底部两行分别为会话和视图快捷键。视频和等待提示从顶部状态栏下方开始，鼠标坐标按视频区域映射，点击状态栏不会传给远端。按键分组使用统一列宽和左右留白，窄窗口自动减少列数、合并按键组并省略次要参数，累计帧数和输出字节数保留在日志中。控制模式下仅提示本地命令入口及后续操作，进入本地命令模式后才显示平移和缩放按键；终端高度不足时优先收起底部提示行。 +画面按字符单元比例等比缩放，留黑边及状态栏。交互连接通过终端字节流查询字符尺寸，支持通过 SSH 返回的回复：优先使用 `CSI 16 t` 的字符像素尺寸，否则组合 `CSI 14 t` 的完整文本区像素尺寸和 `CSI 18 t` 的完整行列数计算，兼容 GNOME/VTE 的文本区报告。计算后再扣除状态栏及画面行列上限，不把预留行当作图像高度。窗口改变时重新查询，每两秒刷新一次以覆盖字体／行距变化；回复到达后会重绘静止画面和更新鼠标映射。查询不阻塞连接；没有有效回复时使用系统提供的像素尺寸，再缺失时回退为宽高比 0.5。曾收到有效回复时保留最近有效尺寸，直到下一次有效回复。字符栅格的整数舍入仍可能产生小幅比例误差。格式依据 [XTerm 窗口尺寸报告](https://invisible-island.net/xterm/ctlseqs/ctlseqs.html) 和 [GNOME VTE 实现](https://github.com/GNOME/vte/blob/master/src/vteseq.cc)。 + +Windows 的可见传统控制台优先读取当前字体尺寸；Windows Terminal／ConPTY 使用显示终端的尺寸回复，不采用隐藏控制台宿主的字体值。不同终端版本可能不支持查询，或返回虚拟图形尺寸而非实际字体尺寸，此时需要手动校准。 + +默认 `--cell-aspect auto` 自动确定比例。如果终端屏蔽查询或需要手动校准，可设置 `--cell-aspect 0.45`，数值表示单个字符的**宽／高**，允许 0.2～2；例如字符为 9×20 像素时使用 0.45。画面被压窄时调小此值，被拉宽时调大。此参数优先于自动检测，同时用于全景、细节和鼠标坐标映射。 + +每次只输出变化行，合并连续颜色设置；缩放在完整输出边界清理旧画面。上限 240 列、100 画面行；默认顶部预留一行状态栏、底部预留两行快捷键，另预留一列避免自动换行。顶部状态栏用反色区分，左侧为控制模式和连接进度，右侧为画面参数及网络信息；底部两行分别为会话和视图快捷键。视频和等待提示从顶部状态栏下方开始，鼠标坐标按视频区域映射，点击状态栏不会传给远端。按键分组使用统一列宽和左右留白，窄窗口自动减少列数、合并按键组并省略次要参数，累计帧数和输出字节数保留在日志中。控制模式下仅提示本地命令入口及后续操作，进入本地命令模式后才显示平移和缩放按键；终端高度不足时优先收起底部提示行。 快捷键行使用 `KEYS |`、`VIEW |` 等固定宽度标题，标题和按键组之间用竖线分隔；键名用 ``、`<+/->` 等标签标识。宽窗口中的分组分隔线按列对齐，窄窗口保留标题和完整按键标签，按可用宽度合并或减少分组。 @@ -123,6 +129,8 @@ pi -e ./apps/cli/plugins/pi/index.ts 在 Pi 输入 `/crossdesk` 或 `/crossdesk 123456789`。先输入 ID，再在自定义遮蔽组件输入密码。扩展使用独立 fd 3 写密码，随后关闭；不写配置、聊天、工具结果或模型上下文。查看组件与独立 CLI 使用相同的 `Ctrl+]` 命令前缀和上述模式按键，由 bridge 统一解析。插件默认请求 high 预览并显示 ANSI 象限字符行，支持同样的只读细节/放大/平移键，颜色深度依据终端环境回退，由 Pi 自身布局和刷新；不直接占用 TTY，不发送 Kitty 图形或自行打开鼠标报告。 +Pi 扩展读取宿主的字符尺寸，并通过宿主终端写入／输入监听接口查询字符尺寸或完整文本区尺寸，每两秒及窗口变化时刷新；尺寸回复不会传给远端。得到的宽高比通过 bridge 的 `viewport.cell_aspect` 传入，比例变化时重绘静止画面，并丢弃仍在途的旧比例帧。自动检测不可用或不准确时，可在启动 Pi 前设置环境变量 `CROSSDESK_CELL_ASPECT=0.45`（宽／高，允许 0.2～2；`auto` 恢复自动检测）。视图关闭或 bridge 停止时移除查询定时器和输入监听。 + 需要全局发现时，把整个 `apps/cli/plugins/pi` 目录复制到 `~/.pi/agent/extensions/crossdesk/` 后重启 Pi 或执行 `/reload`。卸载时先关闭视图，删除该目录并 `/reload`；使用 `-e` 时从启动参数移除即可。无需修改包中的密码或设备配置。默认连接官方信令；自建信令当前通过独立 CLI 连接命令使用，插件尚无服务器设置界面。 仅在 `ctx.mode === "tui"`、hasUI 且进程双向 TTY 时打开组件。RPC/print/json 返回独立播放器提示，不声称内嵌视频。UI 关闭/会话退出先关闭 bridge stdin，超时 SIGTERM，再超时 SIGKILL。bridge 尚未就绪或已经停止时，组件可直接用 `q`、`Esc` 或 `Ctrl+C` 关闭。SDK 类型和模拟 UI 生命周期已验证，真实 Pi TUI 视觉刷新、增强按键与鼠标传递未实机验收。真实设备的 bridge 与独立 CLI 连接验证见 [验证记录](validation.md)。 @@ -144,6 +152,7 @@ pi -e ./apps/cli/plugins/pi/index.ts - `connection_failed` 表示连接建立失败；`connection_lost` 表示连接成功后传输失效。xmake 随后的 `execv ... failed(1)` 表示程序以错误码 1 退出，本身不代表进程崩溃。遇到控制失效时，对照 transport.log 中的控制开关和连接状态；`mouse_reports` 为 0 表示终端没有向 CLI 提供可识别的鼠标事件。报告数增长但 `mouse_forwarded` 为 0 时，检查是否仍为只读／本地命令模式、是否已有画面，以及点击是否落在视频区域；`mouse_send_failures` 增长时检查连接状态。 - 如果整个 macOS Terminal 窗口退出且 CLI 记录 `output closed`，还应检查系统 DiagnosticReports 中的 Terminal 报告。已观察到 Terminal 2.15 在创建标签页并调整窗口尺寸时发生 `TTLogicalScreen expandUnicharOffsetCacheToLine:` 行缓存越界；这与 CLI 的 `connection_failed` 不同。临时可在独立窗口运行，避开该标签页操作，不能据此认为 Terminal 缺陷已修复。 - `doctor --json` 是环境报告，不会联网登录、主动探测 Kitty 或证明远端可连接。 `input_shortcuts=local_commands_v2` 表示支持本文的持续本地命令模式；启动日志同时记录实际执行路径，以便区分不同工程的构建产物。 +- 画面比例异常时查看 `client.log` 摘要的 `terminal_geometry`：`cell_aspect` 为实际采用的字符宽高比，`reported` 表示收到有效的终端尺寸报告，`manual_aspect` 表示使用了手动覆盖。`doctor` 中的比例和像素尺寸只反映系统 TTY 信息，不主动查询终端，可能与交互连接自动检测后的值不同。 第一轮无音频播放或音频订阅 API，SSH 服务器上的声音不会传回本地。无后台守护进程、共享媒体 Broker、IP 直连、MCP 或自动截图。其他宿主可以使用 [私有桥接](bridge-protocol.md)，没有可靠自定义 UI 时打开独立终端或给出播放器命令。 diff --git a/apps/cli/docs/validation.md b/apps/cli/docs/validation.md index a62d0a20..b5479679 100644 --- a/apps/cli/docs/validation.md +++ b/apps/cli/docs/validation.md @@ -58,6 +58,21 @@ - macOS arm64 的 CLI debug 构建和 `doctor --json` 通过。生产 CLI 配合工程外假传输的 POSIX PTY 检查在 120／50 列窗口通过，验证逐字节鼠标输入、`Mouse RX`／`M:` 状态栏计数、日志统计、鼠标报告模式启停和原始终端模式恢复。 - 所有额外测试代码、脚本和产物均位于工程外临时目录。上述结果未覆盖真实 SSH 链路、GNOME Terminal 或 Linux 被控端实机，不能据此确认用户现场的问题已消失。 +## 字符宽高比自动检测（2026-10-30） + +- macOS arm64 CLI debug 构建通过。工程外 ASan/UBSan 检查覆盖字符／文本区尺寸报告、回复顺序与优先级、跨查询部分回复隔离、逐字节输入、本地命令／粘贴隔离、非法尺寸，以及 ANSI half／quadrant／ASCII 的画面比例与鼠标映射。 +- 以 9×20 像素字符、120×40 字符视口和 1920×1080 源图模拟非 0.5 字符比例，旧默认值产生约 8%～9.3% 的宽高比误差；应用实际字符尺寸后为 1.25%，剩余为该网格的整数采样舍入。 +- 生产 CLI 配合工程外假传输的 POSIX PTY 检查通过，覆盖 GNOME 风格的文本区像素／行列回复、直接字符尺寸、字体变化、窗口缩放、无回复／部分回复／非法回复、TTY 回退、手动覆盖、auto 和 Kitty。字体变化时没有新视频帧也能重绘，退出后终端模式恢复。 +- 新增测试及产物均在工程外临时目录；尚未在真实 Linux GNOME Terminal 或 SSH 链路进行视觉验收。 + +## Windows 与 Pi／bridge 字符比例补全（2026-10-30） + +- macOS arm64 CLI debug 构建通过；Windows x64 的主程序、平台 I/O、输入解析和尺寸解析通过交叉编译，平台 I/O 独立链接通过（含 user32／advapi32），不代表 Windows MiniRTC 完整构建或实机运行。 +- 工程外 Win32 API 模拟检查直接使用生产尺寸读取函数，覆盖可见控制台字体尺寸、隐藏 ConPTY 回退、失败／非法字体尺寸与句柄释放，ASan/UBSan 通过。既有尺寸解析 ASan/UBSan 和 POSIX PTY 回归再次通过。 +- 生产 bridge 配合工程外假传输验证可选比例字段保留、有限数值及范围检查、无效请求原子性、实际布局比例、静止画面重绘与帧比例元数据。 +- Pi 1.4.0 SDK 严格类型检查及模拟宿主运行通过：使用真实 SDK 分片组包，覆盖文本区成对回复、字符尺寸优先级、宿主缓存、字体／窗口变化、手动覆盖、旧比例帧丢弃及 ACK、控制键回归、启动失败／异常退出后的监听与定时器清理。 +- 新增验证代码和产物全部位于工程外系统临时目录；尚未在真实 Windows、GNOME／SSH 或 Pi TUI 进行视觉验收。 + ## 实测边界 本地检查不涵盖 Linux / Windows 完整构建、跨平台远程控制、Kitty/SSH 实际终端显示、真实 Pi TUI 或长时间资源占用。五个平台 CI 配置与 npm 包名均不代表已发布或已验收。 diff --git a/apps/cli/main.cpp b/apps/cli/main.cpp index 0975c86c..f1777214 100644 --- a/apps/cli/main.cpp +++ b/apps/cli/main.cpp @@ -2,6 +2,7 @@ #include #include #include +#include #include #include #include @@ -95,7 +96,8 @@ struct Tty { Write(out, end.data(), end.size()); } }; -Viewport Size(int *status_rows = nullptr) { +Viewport Size(int *status_rows = nullptr, + const TerminalMetrics *metrics = nullptr, double cell_aspect = 0) { auto ws = GetTerminalSize(0); const int reserved_rows = std::clamp(int(ws.rows) - 1, 0, 3); if (status_rows) @@ -112,6 +114,16 @@ Viewport Size(int *status_rows = nullptr) { v.cell_aspect = std::clamp(double(ws.pixel_width) * ws.rows / (double(ws.pixel_height) * ws.columns), .2, 2.0); + // A visible Windows console's font metrics are authoritative. ConPTY has + // no local font geometry, so use the display terminal's replies there. +#ifdef _WIN32 + if (metrics && (!ws.pixel_width || !ws.pixel_height)) +#else + if (metrics) +#endif + metrics->Apply(v); + if (cell_aspect > 0) + v.cell_aspect = cell_aspect; auto locale = Env("LC_ALL"); if (locale.empty()) locale = Env("LC_CTYPE"); @@ -172,6 +184,17 @@ bool Boolean(const json &j, const char *key, bool fallback) { throw std::invalid_argument(std::string(key) + " must be boolean"); return j[key]; } +double Real(const json &j, const char *key, double lo, double hi, + double fallback) { + if (!j.contains(key)) + return fallback; + if (!j[key].is_number()) + throw std::invalid_argument(std::string(key) + " must be a number"); + const double value = j[key].get(); + if (!std::isfinite(value) || value < lo || value > hi) + throw std::invalid_argument(std::string(key) + " out of range"); + return value; +} void Quality(SessionOptions &options, const std::string &name) { if (name == "low") { options.quality = 0; @@ -219,7 +242,7 @@ std::string WaitingScreen(const Viewport &v, const std::string &title, } int View(SessionOptions options, const std::string &device, const std::string &renderer, bool control, int fd, AnsiDetail detail, - double zoom, bool native) { + double zoom, bool native, double cell_aspect) { if (!IsTty(0) || !IsTty(fd) || Env("TERM") == "dumb") throw std::runtime_error("connecting needs an interactive ANSI terminal; use " "bridge --stdio for a host"); @@ -257,7 +280,8 @@ int View(SessionOptions options, const std::string &device, std::optional displayed, pending; bool pending_frame = false; int last_status_rows = 3; - Viewport last_view = Size(&last_status_rows); + Viewport last_view = + Size(&last_status_rows, &input.terminal_metrics, cell_aspect); std::optional last_frame; uint64_t view_revision = input.view_revision; bool redraw = true; @@ -267,6 +291,7 @@ int View(SessionOptions options, const std::string &device, bool waiting_visible = false; double queue_ms = 0; auto next = Clock::now(), status_tick = next; + auto size_query_tick = next; const auto pump_output = [&] { if (output.Pump() && pending_frame) { displayed = std::move(pending); @@ -300,7 +325,7 @@ int View(SessionOptions options, const std::string &device, break; auto now = Clock::now(); int status_rows; - auto v = Size(&status_rows); + auto v = Size(&status_rows, &input.terminal_metrics, cell_aspect); v.detail = detail; v.native = input.native; v.pixel_mode = kitty; @@ -311,8 +336,10 @@ int View(SessionOptions options, const std::string &device, view_revision = input.view_revision; redraw = true; } + const bool grid_resized = + v.columns != last_view.columns || v.rows != last_view.rows; if (output.Idle() && - (v.columns != last_view.columns || v.rows != last_view.rows || + (grid_resized || v.cell_aspect != last_view.cell_aspect || v.pixel_width != last_view.pixel_width || v.pixel_height != last_view.pixel_height || status_rows != last_status_rows)) { @@ -322,6 +349,15 @@ int View(SessionOptions options, const std::string &device, last_status_rows = status_rows; redraw = true; status_tick = now; + if (grid_resized) + size_query_tick = now; + } + // Query through the terminal byte stream so this also works over SSH. + // Never wait for a reply or interrupt a partially written video frame. + if (output.Idle() && now >= size_query_tick) { + input.terminal_metrics.BeginQuery(); + output.Put(kTerminalSizeQuery); + size_query_tick = now + std::chrono::seconds(2); } if (output.Idle() && now >= status_tick) { auto st = s.Status(); @@ -449,6 +485,12 @@ int View(SessionOptions options, const std::string &device, {"mouse_legacy_reports", input.mouse_legacy_reports}, {"mouse_forwarded", input.mouse_forwarded}, {"mouse_send_failures", input.mouse_send_failures}, + {"terminal_geometry", + {{"cell_aspect", last_view.cell_aspect}, + {"pixel_width", last_view.pixel_width}, + {"pixel_height", last_view.pixel_height}, + {"reported", input.terminal_metrics.Known()}, + {"manual_aspect", cell_aspect > 0}}}, {"submitted", submitted}, {"output_bytes", output.bytes}, {"last_queue_ms", queue_ms}, @@ -468,15 +510,16 @@ int Bridge(int fd, SessionOptions options, int password_fd) { Output output(fd); ControllerSession s; Input input; + Viewport viewport; + viewport.detail = AnsiDetail::Quadrant; auto status = [&] { auto value = s.Status(); value["command_mode"] = input.command_mode; value["resume_control"] = input.control; value["exit_requested"] = input.exit; + value["cell_aspect"] = viewport.cell_aspect; return value; }; - Viewport viewport; - viewport.detail = AnsiDetail::Quadrant; NonBlocking input_mode(0); std::deque replies; std::string buffer; @@ -548,6 +591,8 @@ int Bridge(int fd, SessionOptions options, int password_fd) { next_view.colors != 256) throw std::invalid_argument("unsupported_colors"); next_view.utf8 = Boolean(j, "utf8", true); + next_view.cell_aspect = + Real(j, "cell_aspect", .2, 2, viewport.cell_aspect); if (j.contains("detail")) next_view.detail = Detail(j.at("detail").get()); next_options.fps = Integer(j, "fps", 1, 30, 8); @@ -687,6 +732,7 @@ int Bridge(int fd, SessionOptions options, int password_fd) { {"generation", last_frame->generation}, {"columns", viewport.columns}, {"rows", viewport.rows}, + {"cell_aspect", viewport.cell_aspect}, {"source_width", last_frame->width}, {"source_height", last_frame->height}, {"zoom", viewport.zoom}, @@ -718,6 +764,7 @@ int Bridge(int fd, SessionOptions options, int password_fd) { const char *help = R"(crossdesk-cli <9-digit-device-id> [--renderer auto|ansi|kitty] [--fps 1..30] [--display 0..63] [--control] [--server host] [--port 9099] [--quality low|balanced|high|ultra] [--ansi-detail quadrant|half] [--zoom 1..32|native] + [--cell-aspect auto|0.2..2] crossdesk-cli view <9-digit-device-id> [options] (compatibility alias) crossdesk-cli doctor [--json] crossdesk-cli bridge --stdio [--password-fd 3] [--server host] [--port 9099] @@ -729,6 +776,7 @@ View keys: click/d native detail, wheel/+/- zoom, arrows/ijkl pan (i up, j left, Local mode stays active after commands, paste or focus loss; remote input remains paused. Ctrl+T and Ctrl+C pass through in control mode; another Ctrl+] leaves local mode and forwards Ctrl+] if control is enabled. Default ANSI detail: quadrant. Default quality: high (1920x1080); ultra requests up to 3840x2160. +Cell aspect is character width/height: auto detects terminal geometry; use --cell-aspect to calibrate. Display index is zero-based. ANSI cannot reproduce full desktop text at fit scale. --fps controls local output; negotiated previews use supported 1/8/15 FPS. bridge: private NDJSON v1, credentials on a separate inherited descriptor. @@ -758,6 +806,9 @@ int main(int argc, char **argv) { {"transport_diagnostics", true}, {"utf8", v.utf8}, {"colors", v.colors}, + {"cell_aspect", v.cell_aspect}, + {"pixel_width", v.pixel_width}, + {"pixel_height", v.pixel_height}, {"multiplexer", !Env("TMUX").empty() || !Env("STY").empty()}, {"software_decode", true}, {"kitty", "probe_on_view"}, @@ -777,7 +828,7 @@ int main(int argc, char **argv) { SessionOptions options; std::string renderer = "auto", device; AnsiDetail detail = AnsiDetail::Quadrant; - double zoom = 1; + double zoom = 1, cell_aspect = 0; bool control = false, stdio = false, native = false; int secret_fd = 3; if (command == "view") { @@ -819,7 +870,17 @@ int main(int argc, char **argv) { Quality(options, value); else if (a == "--ansi-detail" && command == "view") detail = Detail(value); - else if (a == "--zoom" && command == "view") { + else if (a == "--cell-aspect" && command == "view") { + if (value == "auto") { + cell_aspect = 0; + } else { + size_t used = 0; + cell_aspect = std::stod(value, &used); + if (used != value.size() || !std::isfinite(cell_aspect) || + cell_aspect < .2 || cell_aspect > 2) + throw std::invalid_argument("cell aspect must be auto or 0.2..2"); + } + } else if (a == "--zoom" && command == "view") { native = value == "native"; zoom = native ? 1 : number(value, 1, 32); } else if (a == "--password-fd" && command == "bridge") @@ -854,7 +915,8 @@ int main(int argc, char **argv) { << "\n"; try { return command == "view" ? View(options, device, renderer, control, - output.value, detail, zoom, native) + output.value, detail, zoom, native, + cell_aspect) : Bridge(output.value, options, secret_fd); } catch (const std::exception &e) { if (command == "view") { diff --git a/apps/cli/plugins/pi/index.ts b/apps/cli/plugins/pi/index.ts index 34d566b4..b1ea699c 100644 --- a/apps/cli/plugins/pi/index.ts +++ b/apps/cli/plugins/pi/index.ts @@ -1,9 +1,14 @@ import type { ExtensionAPI } from "@earendil-works/pi-coding-agent"; -import { truncateToWidth, parseKey, decodeKittyPrintable, isKeyRelease } from "@earendil-works/pi-tui"; +import { truncateToWidth, parseKey, decodeKittyPrintable, isKeyRelease, getCellDimensions } from "@earendil-works/pi-tui"; import { spawn, type ChildProcess } from "node:child_process"; import type { Writable } from "node:stream"; const clean = (s: unknown) => String(s ?? "").replace(/[^\x20-\x7e]/g, "?").slice(0, 180); +const validAspect = (n: number) => Number.isFinite(n) && n >= 0.2 && n <= 2; +function aspectOf(width: number, height: number): number | undefined { + const ratio = width / height; + return width >= 1 && width <= 1024 && height >= 1 && height <= 1024 && validAspect(ratio) ? ratio : undefined; +} // Normalize enhanced key reports with the host's actual keyboard decoder. function legacyInput(data: string): string { @@ -29,6 +34,12 @@ export default function crossdesk(pi: ExtensionAPI) { return; } if (stop) { ctx.ui.notify("A CrossDesk view is already open", "warning"); return; } + const configuredAspect = process.env.CROSSDESK_CELL_ASPECT?.trim(); + const manualAspect = !configuredAspect || configuredAspect === "auto" ? undefined : Number(configuredAspect); + if (manualAspect !== undefined && !validAspect(manualAspect)) { + ctx.ui.notify("CROSSDESK_CELL_ASPECT must be auto or a width/height ratio from 0.2 to 2", "warning"); + return; + } const device = (args.trim() || await ctx.ui.input("CrossDesk device ID", "9 digits"))?.trim(); if (!device) return; if (!/^\d{9}$/.test(device)) { ctx.ui.notify("Expected a 9-digit device ID", "error"); return; } @@ -56,7 +67,9 @@ export default function crossdesk(pi: ExtensionAPI) { if (password === undefined) return; let child: ChildProcess | undefined; let killTimer: ReturnType | undefined; + let stopGeometry: (() => void) | undefined; const cleanup = () => { + stopGeometry?.(); stopGeometry = undefined; if (!child) return; const owned = child; child = undefined; owned.stdin?.end(); // EOF asks the bridge to disconnect and release input. @@ -75,6 +88,8 @@ export default function crossdesk(pi: ExtensionAPI) { let frame: { frame_id: number; generation: number } | undefined; let helloId = 0, zoom = 1, native = false; let control = false, commandMode = false, bridgeStopped = false; + const hostCell = getCellDimensions(); + let cellAspect = manualAspect ?? aspectOf(hostCell.widthPx, hostCell.heightPx) ?? 0.5; const colorHint = process.env.COLORTERM || ""; const colors = /^(truecolor|24bit)$/.test(colorHint) ? 24 : process.env.TERM?.includes("256color") ? 256 : 8; const finish = () => { if (closed) return; closed = true; cleanup(); stop = undefined; lines = []; done(); }; @@ -90,13 +105,21 @@ export default function crossdesk(pi: ExtensionAPI) { const id = ++request; child.stdin.write(JSON.stringify({ id, method, ...fields }) + "\n"); return id; }; + const sendViewport = () => send("viewport", { + columns: columns || 80, rows: rows || 20, cell_aspect: cellAspect, + colors, utf8: true, fps: 8, detail: "quadrant", quality: "high", + }); child.stdin?.on("error", () => fail("Bridge input closed")); const secrets = child.stdio[3] as Writable; secrets.on("error", () => fail("Credential channel closed")); secrets.end(password + "\n"); password = undefined; child.stderr?.resume(); // Never insert native diagnostics in the TUI or chat. child.on("error", () => fail("Cannot start crossdesk-cli; check CROSSDESK_CLI or PATH")); - child.on("exit", (code) => { if (!closed) { bridgeStopped = true; error ||= `Bridge exited (${code ?? "signal"})`; tui.requestRender(); } }); + child.on("exit", (code) => { + if (child === processChild) child = undefined; + stopGeometry?.(); stopGeometry = undefined; + if (!closed) { bridgeStopped = true; error ||= `Bridge exited (${code ?? "signal"})`; tui.requestRender(); } + }); child.stdout?.setEncoding("utf8"); child.stdout?.on("data", (data: string) => { if (closed) return; @@ -109,7 +132,7 @@ export default function crossdesk(pi: ExtensionAPI) { const message = JSON.parse(raw); if (message.id === helloId && message.ok && message.version === 1 && !ready) { ready = true; - send("viewport", { columns: columns || 80, rows: rows || 20, colors, utf8: true, fps: 8, detail: "quadrant", quality: "high" }); + sendViewport(); send("connect", { device }); } if (message.ok === false) error = clean(message.error); @@ -128,7 +151,8 @@ export default function crossdesk(pi: ExtensionAPI) { if (!Number.isSafeInteger(message.frame_id) || !Number.isSafeInteger(message.generation) || !Array.isArray(message.lines) || message.lines.length > 100) throw new Error("Invalid frame"); - if (message.columns !== columns || message.rows !== rows || message.generation < generation) { + if (message.columns !== columns || message.rows !== rows || message.generation < generation || + (message.cell_aspect !== undefined && message.cell_aspect !== cellAspect)) { send("ack", {frame_id: message.frame_id, generation: message.generation}); continue; // A frame already in flight may precede the resize response. } @@ -148,13 +172,57 @@ export default function crossdesk(pi: ExtensionAPI) { helloId = send("hello", { version: 1 }); const watchdog = setTimeout(() => { if (!ready) fail("Bridge handshake timeout"); }, 5000); processChild.once("exit", () => clearTimeout(watchdog)); + // Use the host's terminal writer and sequence listener; Pi remains + // the sole owner of raw input and screen rendering. + let pixelSize: [number, number] | undefined, cellCount: [number, number] | undefined; + let directCell = false, queried = false; + const updateAspect = (width: number, height: number) => { + const ratio = aspectOf(width, height); + if (manualAspect !== undefined || ratio === undefined || ratio === cellAspect) return; + cellAspect = ratio; lines = []; frame = undefined; + if (ready) sendViewport(); + tui.requestRender(); + }; + const removeGeometryListener = tui.addInputListener(data => { + // Pi's StdinBuffer assembles split replies before listeners run. + const match = /^\x1b\[([468]);([0-9]+);([0-9]+)t$/.exec(data); + if (!queried || !match) return; + const type = Number(match[1]), height = Number(match[2]), width = Number(match[3]); + if (![height, width].every(n => Number.isSafeInteger(n) && n > 0 && n <= 1048576)) + return { consume: true }; + if (type === 6) { + if (aspectOf(width, height) === undefined) return { consume: true }; + directCell = true; + updateAspect(width, height); + return; // Let Pi update its own shared cell-size cache as well. + } + if (type === 4) pixelSize = [width, height]; + if (type === 8) cellCount = [width, height]; + if (!directCell && pixelSize && cellCount) + updateAspect(pixelSize[0] / cellCount[0], pixelSize[1] / cellCount[1]); + return { consume: true }; + }); + let terminalColumns = tui.terminal.columns, terminalRows = tui.terminal.rows; + const queryGeometry = () => { + if (closed || bridgeStopped) return; + queried = true; directCell = false; pixelSize = cellCount = undefined; + tui.terminal.write("\x1b[16t\x1b[14t\x1b[18t"); + }; + const geometryTimer = setInterval(queryGeometry, 2000); + geometryTimer.unref(); + stopGeometry = () => { clearInterval(geometryTimer); removeGeometryListener(); }; + queryGeometry(); return { render(width: number) { const w = Math.max(1, Math.min(240, width - 1)); const h = Math.max(1, Math.min(100, tui.terminal.rows - 5)); if (w !== columns || h !== rows) { columns = w; rows = h; lines = []; frame = undefined; - if (ready) send("viewport", { columns, rows, colors, utf8: true, fps: 8, detail: "quadrant", quality: "high" }); + if (ready) sendViewport(); + } + if (terminalColumns !== tui.terminal.columns || terminalRows !== tui.terminal.rows) { + terminalColumns = tui.terminal.columns; terminalRows = tui.terminal.rows; + queryGeometry(); } const ack = frame; frame = undefined; if (ack) queueMicrotask(() => send("ack", ack)); // Host consumed the frame; not a physical display timestamp. diff --git a/apps/cli/terminal_io_windows.cpp b/apps/cli/terminal_io_windows.cpp index b5f00d93..cc0b9142 100644 --- a/apps/cli/terminal_io_windows.cpp +++ b/apps/cli/terminal_io_windows.cpp @@ -312,6 +312,19 @@ TerminalSize GetTerminalSize(int) { if (GetConsoleScreenBufferInfo(handle, &info)) { size.columns = info.srWindow.Right - info.srWindow.Left + 1; size.rows = info.srWindow.Bottom - info.srWindow.Top + 1; + // ConPTY's console window is only a message window, and its font API + // describes the hidden console host, not the user's display terminal. + // Only trust native font metrics for a visible classic console. + HWND window = GetConsoleWindow(); + CONSOLE_FONT_INFOEX font{}; + font.cbSize = sizeof(font); + if (window && IsWindowVisible(window) && + GetCurrentConsoleFontEx(handle, FALSE, &font) && + font.dwFontSize.X > 0 && font.dwFontSize.X <= 1024 && + font.dwFontSize.Y > 0 && font.dwFontSize.Y <= 1024) { + size.pixel_width = size.columns * font.dwFontSize.X; + size.pixel_height = size.rows * font.dwFontSize.Y; + } } if (console_output < 0 && handle != INVALID_HANDLE_VALUE) CloseHandle(handle); diff --git a/apps/cli/xmake.lua b/apps/cli/xmake.lua index 4fa30c48..3faf9cee 100644 --- a/apps/cli/xmake.lua +++ b/apps/cli/xmake.lua @@ -42,7 +42,7 @@ target("crossdesk-cli") add_files("main.cpp") add_files(is_plat("windows") and "terminal_io_windows.cpp" or "terminal_io_posix.cpp") if is_plat("windows") then - add_syslinks("advapi32") + add_syslinks("advapi32", "user32") add_defines("NOMINMAX", "WIN32_LEAN_AND_MEAN") end target_end() diff --git a/libs/terminal/include/terminal_renderer.h b/libs/terminal/include/terminal_renderer.h index 2337054d..2229952b 100644 --- a/libs/terminal/include/terminal_renderer.h +++ b/libs/terminal/include/terminal_renderer.h @@ -21,6 +21,22 @@ struct Viewport { bool native = false, pixel_mode = false; int row_offset = 0; // Terminal rows reserved above the video area. }; +// Keep reported cell geometry across queries; never combine partial replies +// from different query cycles. Pixel/cell window sizes use the full text area. +class TerminalMetrics { +public: + void BeginQuery(); + void Report(const std::string &sequence); + void Apply(Viewport &) const; + bool Known() const { return cell_width_ > 0; } + +private: + bool SetCellSize(double width, double height); + double cell_width_ = 0, cell_height_ = 0; + int pixel_width_ = 0, pixel_height_ = 0, columns_ = 0, rows_ = 0; + bool queried_ = false, direct_reply_ = false; +}; +inline constexpr char kTerminalSizeQuery[] = "\033[16t\033[14t\033[18t"; struct Rect { int x, y, width, height; }; @@ -72,6 +88,7 @@ public: uint64_t mouse_reports = 0; uint64_t mouse_legacy_reports = 0, mouse_forwarded = 0; uint64_t mouse_send_failures = 0; + TerminalMetrics terminal_metrics; private: std::string pending_; diff --git a/libs/terminal/src/input.cpp b/libs/terminal/src/input.cpp index 6c21cd88..4f89e1ea 100644 --- a/libs/terminal/src/input.cpp +++ b/libs/terminal/src/input.cpp @@ -238,6 +238,10 @@ void Input::Feed(const std::string &bytes, ControllerSession &s, return; std::string seq = pending_.substr(0, end + 1); pending_.erase(0, end + 1); + if (seq.back() == 't') { + terminal_metrics.Report(seq); + continue; + } if (seq == "\033[O") { control = false; s.SetControl(false); diff --git a/libs/terminal/src/terminal_metrics.cpp b/libs/terminal/src/terminal_metrics.cpp new file mode 100644 index 00000000..60834a85 --- /dev/null +++ b/libs/terminal/src/terminal_metrics.cpp @@ -0,0 +1,67 @@ +#include "terminal_renderer.h" +#include +#include + +namespace crossdesk::terminal { +void TerminalMetrics::BeginQuery() { + queried_ = true; + direct_reply_ = false; + pixel_width_ = pixel_height_ = columns_ = rows_ = 0; +} + +bool TerminalMetrics::SetCellSize(double width, double height) { + if (width < 1 || height < 1 || width > 1024 || height > 1024 || + width / height < .2 || width / height > 2) + return false; + cell_width_ = width; + cell_height_ = height; + return true; +} + +void TerminalMetrics::Report(const std::string &sequence) { + if (!queried_ || sequence.size() < 9 || sequence.compare(0, 2, "\033[") || + sequence.back() != 't') + return; + int type = 0, height = 0, width = 0; + const char *cur = sequence.data() + 2; + const char *end = sequence.data() + sequence.size() - 1; + for (int *value : {&type, &height, &width}) { + auto result = std::from_chars(cur, end, *value); + if (result.ec != std::errc{} || *value <= 0 || *value > 1048576) + return; + cur = result.ptr; + if (value != &width) { + if (cur == end || *cur++ != ';') + return; + } else if (cur != end) { + return; + } + } + if (type == 6) { + // CSI 16 t replies with character height, then width (not width/height). + if (SetCellSize(width, height)) + direct_reply_ = true; + } else if (type == 4) { + pixel_width_ = width; + pixel_height_ = height; + } else if (type == 8) { + columns_ = width; + rows_ = height; + } else { + return; + } + // VTE supports text-area pixels/cells even when CSI 16 t is unavailable. + // Use reported full rows/columns, not the capped video viewport or status + // rows, and retain fractional cell dimensions for scaled terminal fonts. + if (!direct_reply_ && pixel_width_ && pixel_height_ && columns_ && rows_) + SetCellSize(double(pixel_width_) / columns_, double(pixel_height_) / rows_); +} + +void TerminalMetrics::Apply(Viewport &viewport) const { + if (!Known()) + return; + viewport.cell_aspect = cell_width_ / cell_height_; + viewport.pixel_width = int(std::lround(cell_width_ * viewport.columns)); + viewport.pixel_height = int(std::lround(cell_height_ * viewport.rows)); +} +} // namespace crossdesk::terminal