diff --git a/apps/cli/docs/bridge-protocol.md b/apps/cli/docs/bridge-protocol.md index d73ddb37..1ed4a355 100644 --- a/apps/cli/docs/bridge-protocol.md +++ b/apps/cli/docs/bridge-protocol.md @@ -20,6 +20,8 @@ 凭据不能放在 connect JSON 中。专用描述符可以由 Node `spawn` 的第四条 pipe 提供。成功读取后会话不会保留用户密码供上层访问;MiniRTC 内部为现有 Join/重连流程持有必要凭据直到销毁 Peer。下一次 connect 必须有可读的凭据来源;Pi 关闭视图后直接结束进程。 +`viewport.fps` 是本地输出目标。对端支持预览订阅时,超过 15 的目标使用已有交互模式(协议 `fps=60`、编码 `frame_rate=30`),不再折算成 15 FPS;低目标仍按能力选择 1/8/15。bridge 始终输出字符行并要求宿主 ACK,不使用交互 Kitty 的共享内存传图。 + 帧事件: ```json diff --git a/apps/cli/docs/terminal-client.md b/apps/cli/docs/terminal-client.md index c2538d35..80879634 100644 --- a/apps/cli/docs/terminal-client.md +++ b/apps/cli/docs/terminal-client.md @@ -45,13 +45,17 @@ crossdesk-cli 123456789 --server your-server.example --port 9099 ANSI 默认使用四象限字符:每个字符从 2×2 样本选择误差最小的两个颜色和象限形状,比原先上下半方块多一倍水平采样点;每个字符仍只有前景/背景两色。使用 `--ansi-detail half` 可回到上下半方块以兼容字体。缩小时采用面积加权滤波,避免最近邻采样漏掉细线。支持 TrueColor、256 色和基础 8 色,256 色选择实际颜色立方体或灰阶条中更接近的颜色。环境的 COLORTERM/TERM 决定颜色模式;非 UTF-8 locale 使用背景色空格。TERM=dumb、非 TTY 输出明确拒绝交互连接,不输出全屏转义序列。 +四象限模式将每个 RGB 通道变化均不超过 6/255 的低对比单元合并为纯背景色,减少平坦区域的字形接缝。其余单元在独立拟合后参考上下左右邻格的量化误差调整选色,保留源图的真实渐变,并在高对比边缘减弱约束;调整设有失真上限,超过低对比阈值且能精确表示的文字细线和双色边缘保持原样。这些处理自动应用于独立 CLI 和 Pi/bridge 的四象限画面,实际效果仍受终端字体和行距影响。 + 画面按字符单元比例等比缩放,留黑边及状态栏。交互连接通过终端字节流查询字符尺寸,支持通过 SSH 返回的回复:优先使用 `CSI 16 t` 的字符像素尺寸,否则组合 `CSI 14 t` 的完整文本区像素尺寸和 `CSI 18 t` 的完整行列数计算,兼容 GNOME/VTE 的文本区报告。计算后再扣除状态栏及画面行列上限,不把预留行当作图像高度。窗口改变时重新查询,每两秒刷新一次以覆盖字体/行距变化;回复到达后会重绘静止画面和更新鼠标映射。查询不阻塞连接;没有有效回复时使用系统提供的像素尺寸,再缺失时回退为宽高比 0.5。曾收到有效回复时保留最近有效尺寸,直到下一次有效回复。字符栅格的整数舍入仍可能产生小幅比例误差。格式依据 [XTerm 窗口尺寸报告](https://invisible-island.net/xterm/ctlseqs/ctlseqs.html) 和 [GNOME VTE 实现](https://github.com/GNOME/vte/blob/master/src/vteseq.cc)。 Windows 的可见传统控制台优先读取当前字体尺寸;Windows Terminal/ConPTY 使用显示终端的尺寸回复,不采用隐藏控制台宿主的字体值。不同终端版本可能不支持查询,或返回虚拟图形尺寸而非实际字体尺寸,此时需要手动校准。 默认 `--cell-aspect auto` 自动确定比例。如果终端屏蔽查询或需要手动校准,可设置 `--cell-aspect 0.45`,数值表示单个字符的**宽/高**,允许 0.2~2;例如字符为 9×20 像素时使用 0.45。画面被压窄时调小此值,被拉宽时调大。此参数优先于自动检测,同时用于全景、细节和鼠标坐标映射。 -每次只输出变化行,合并连续颜色设置;缩放在完整输出边界清理旧画面。上限 240 列、100 画面行;默认顶部预留一行状态栏、底部预留两行快捷键,另预留一列避免自动换行。顶部状态栏用反色区分,左侧为控制模式和连接进度,右侧为画面参数及网络信息;底部两行分别为会话和视图快捷键。视频和等待提示从顶部状态栏下方开始,鼠标坐标按视频区域映射,点击状态栏不会传给远端。按键分组使用统一列宽和左右留白,窄窗口自动减少列数、合并按键组并省略次要参数,累计帧数和输出字节数保留在日志中。控制模式下仅提示本地命令入口及后续操作,进入本地命令模式后才显示平移和缩放按键;终端高度不足时优先收起底部提示行。 +独立 CLI 每次只输出变化行中首尾变化单元之间的区间,光标或局部视频更新不再重写整行;一帧内复用颜色状态、按实际色板索引合并颜色设置,并省略纯背景空格的前景色指令。Pi/bridge 仍发送可独立绘制的完整字符行,同时使用上述颜色编码优化。四象限初始拟合使用整数子集求和计算同等精度的颜色和误差,边界连续性拟合复用加权样本,减少重复运算。 + +缩放在完整输出边界清理旧画面。上限 240 列、100 画面行;默认顶部预留一行状态栏、底部预留两行快捷键,另预留一列避免自动换行。顶部状态栏用反色区分,左侧为控制模式和连接进度,右侧为画面参数及网络信息;底部两行分别为会话和视图快捷键。视频和等待提示从顶部状态栏下方开始,鼠标坐标按视频区域映射,点击状态栏不会传给远端。按键分组使用统一列宽和左右留白,窄窗口自动减少列数、合并按键组并省略次要参数,累计帧数和输出字节数保留在日志中。控制模式下仅提示本地命令入口及后续操作,进入本地命令模式后才显示平移和缩放按键;终端高度不足时优先收起底部提示行。 快捷键行使用 `KEYS |`、`VIEW |` 等固定宽度标题,标题和按键组之间用竖线分隔;键名用 ``、`<+/->` 等标签标识。宽窗口中的分组分隔线按列对齐,窄窗口保留标题和完整按键标签,按可用宽度合并或减少分组。 @@ -59,7 +63,11 @@ ANSI 在完整桌面缩略图中仍无法保留普通字号的文字。先增加 auto 模式发送 Kitty 查询与设备属性查询,等待最多 250ms;没有有效 Kitty 回应就用 ANSI。显式 kitty 模式探测失败时报错。tmux/screen 中 auto 使用 ANSI,显式 kitty 报不支持;不修改用户终端配置。 -Kitty 通过终端字节流发送 zlib 压缩的内联 RGB/Base64(每块至多 4096 字符),不使用远端临时图片文件。图像固定 ID=31、placement=1,更新同一图像,退出删除。依据终端报告的像素尺寸生成画面(未知时按 10×20 像素字符估算),上限 1920×1080。保留黑边和宽高比;原始细节模式按像素视口裁取源区域,避免整桌面缩小。没有可靠终端像素尺寸时使用估算值,因此不承诺显示设备的物理像素一一对应。压缩后超出单次输出预算时逐级降低尺寸。高熵视频压缩率较低,同一输出预算下实际帧率可能低于目标。RGB 压缩无损,不代表上游视频编码无损。 +本地 macOS/Linux 的 Kitty 模式会另用一个 1×1 像素的共享内存对象探测 `t=s`,最多等待 250ms;探测成功后,将未压缩 RGB 写入权限为 0600 的 POSIX 共享内存,仅通过终端字节流发送名字和显示命令。每次使用唯一名字,最多保留一个待确认的共享缓冲区(1920×1080×3 字节)。收到上传成功确认后才显示新图并释放缓冲区;上传报错、确认超过 500ms 或本地分配失败时,本次会话退回内联传输,继续保留旧画面。终端和 CLI 都会清理共享对象,退出时也会释放未确认的对象。SSH 中不尝试共享内存;Windows 暂用内联传输。 + +内联 Kitty 通过终端字节流发送 zlib 压缩的 RGB/Base64(每块至多 4096 字符),不使用远端临时图片文件。两条路径都交替使用图像 ID=31/32:以 `a=t` 上传到未显示的 ID,传完后以 `a=p` 创建 placement=1,再删除旧图。仅在整帧显示命令输出完成后交替 ID,上传和短写期间保留旧画面,退出时清理两个 ID。这样避免重传同一 ID 时终端先移除旧图造成的帧间黑屏。 + +依据终端报告的像素尺寸生成画面(未知时按 10×20 像素字符估算),上限 1920×1080。保留面积滤波、黑边、宽高比和裁剪坐标;原始细节模式按像素视口裁取源区域。共享内存路径使用可复用水平行计算的整数面积权重,避免逐像素重复浮点运算,也不再因 Base64 字节预算额外降采样。没有可靠终端像素尺寸时使用估算值,因此不承诺显示设备的物理像素一一对应。内联压缩结果超出单次输出预算时仍逐级降低尺寸;高熵视频的实际帧率可能低于目标。RGB 传输和压缩不丢失像素,不代表面积采样或上游视频编码无损。 ```sh ssh -t your-server @@ -70,7 +78,7 @@ crossdesk-cli 123456789 --renderer ansi --fps 8 ## 预览与背压 -`--fps 1..30` 是本地输出目标。对端声明 PreviewSubscription 时,映射到合法 1/8/15 FPS(15 需要对应能力);普通编码设置仍使用合法 30 FPS。分辨率上限使用协商到的 v2,视频预算只在 bandwidth 能力存在时发送。`--quality low|balanced|high|ultra` 对应如下请求: +`--fps 1..30` 是本地输出目标。未显式指定时,共享内存探测成功默认使用 30 FPS;ANSI、内联 Kitty 和 bridge 默认仍为 8 FPS。自动选择的共享内存路径运行中退回内联时,目标也恢复为 8;显式设置的 `--fps` 始终保留。对端声明 PreviewSubscription 时,目标不超过 15 映射到合法 1/8/15 FPS(15 需要对应能力);超过 15 则请求项目已有的交互模式:协议 `fps=60` 是模式标记,编码设置 `frame_rate=30` 才是帧率,不再把 `--fps 30` 限制为 15 FPS。分辨率上限使用协商到的 v2,视频预算只在 bandwidth 能力存在时发送。`--quality low|balanced|high|ultra` 对应如下请求: | 档位 | 分辨率上限 | 质量枚举 | 视频预算 | |---|---|---|---| @@ -83,9 +91,11 @@ crossdesk-cli 123456789 --renderer ansi --fps 8 媒体回调仅同步复制借用的 NV12 数据,校验尺寸、容量、native 描述符与 stride;不保留借用指针。上限单帧 16,777,216 像素、每边 8192、偶数尺寸。最多一个待转换帧,替换前先完成校验与复制,无效帧不会挤掉已有画面;播放器另保留最近转换的一帧用于静止桌面的本地放大/重绘,并在接收新帧时回收上一帧缓冲区供后续复制使用。只丢弃解码后图像。显示器切换/关闭提升 generation,旧帧失效。 -转换与输出在拥有会话的事件循环,媒体回调独立。输出至多一个完整消息(2MiB),以 1MiB/s 令牌预算分块非阻塞写入;不能开始新画面时保留最新解码帧。ANSI 差分基准只在整帧输出完成后提交。Pi 桥接另要求消费帧 ACK,最多一帧等待 ACK,输入映射只在对应 ACK 后切换到新网格。状态栏优先在完整画面输出之间更新,避免大图持续刷新时状态饥饿;可写事件唤醒短写重试,仍遵守字节预算。控制事件至多 64 条,协议输入 64KiB/嵌套 32 层,输入粘贴缓冲 32KiB,文本单次最多 16KiB。 +转换与输出在拥有会话的事件循环,媒体回调独立。输出至多一个完整消息(2MiB),按令牌预算分块非阻塞写入,突发额度上限仍为 1MiB。本地 Kitty 使用 8MiB/s;存在 `SSH_CONNECTION`、`SSH_CLIENT` 或 `SSH_TTY` 时,Kitty 保持 1MiB/s,ANSI 和 Pi 桥接也保持 1MiB/s。不能开始新画面时保留最新解码帧。ANSI 差分基准只在整帧输出完成后提交。Pi 桥接另要求消费帧 ACK,最多一帧等待 ACK,输入映射只在对应 ACK 后切换到新网格。状态栏优先在完整画面输出之间更新,避免大图持续刷新时状态饥饿;可写事件唤醒短写重试,仍遵守字节预算。控制事件至多 64 条,协议输入 64KiB/嵌套 32 层,输入粘贴缓冲 32KiB,文本单次最多 16KiB。 -统计包括收到帧数、被替换帧数、非法帧、等待帧数、实际完整提交次数、输出字节数、运行秒数、最近本地排队时间、网络码率/RTT/SRTP。通过计数差/时间差可得到区间 FPS;交互连接结束日志另给平均提交 FPS。`client.log` 摘要的 `startup_ms` 记录从发起连接到连接成功(`connected`)、首个取出用于显示的视频帧入队(`first_frame`)、首帧完整写入终端(`first_output`)的毫秒数;未到达的阶段为 `null`。写入 TTY 完成不等于屏幕显示完成,本地排队时间不是端到端延迟。 +测量像素渲染帧率时使用 release 构建;debug 的 `-O0` 面积采样开销明显更高。共享内存中的像素不经过终端字节预算,但仍受转换和终端消费速度限制;8MiB/s 是内联路径的输出上限,不是帧率保证。解码、像素转换、压缩、终端消费速度和帧大小都会影响结果。退出时,未完成画面最多继续写入 2s;取消图像传输及恢复屏幕的短写最多另重试 500ms。POSIX 立即恢复 termios 并清理输入,不等待输出排空;如果终端始终不消费输出,屏幕清理只能尽力发送。 + +统计包括收到帧数、被替换帧数、非法帧、等待帧数、实际完整提交次数、输出字节数、运行秒数、最近本地排队时间、网络码率/RTT/SRTP。通过计数差/时间差可得到区间 FPS;交互连接结束日志另给平均提交 FPS,以及实际 `renderer` 和 `output_bytes_per_second` 预算。`kitty_transport=shared_memory/inline/none` 标明生效路径,`target_fps` 是当前目标,`frame_processing_ms` 是平均本地布局/转换/传输准备耗时,`shared_ack_ms` 是平均上传命令写完到解析成功确认的耗时,`shared_fallbacks` 记录运行时回退次数。`client.log` 摘要的 `startup_ms` 记录从发起连接到连接成功(`connected`)、首个取出用于显示的视频帧入队(`first_frame`)、首帧显示命令完整写入终端(`first_output`)的毫秒数;未到达的阶段为 `null`。写入 TTY 完成不等于屏幕显示完成,本地排队时间不是端到端延迟。 ## 输入与退出 diff --git a/apps/cli/docs/validation.md b/apps/cli/docs/validation.md index b5479679..e18f58b2 100644 --- a/apps/cli/docs/validation.md +++ b/apps/cli/docs/validation.md @@ -73,6 +73,45 @@ - Pi 1.4.0 SDK 严格类型检查及模拟宿主运行通过:使用真实 SDK 分片组包,覆盖文本区成对回复、字符尺寸优先级、宿主缓存、字体/窗口变化、手动覆盖、旧比例帧丢弃及 ACK、控制键回归、启动失败/异常退出后的监听与定时器清理。 - 新增验证代码和产物全部位于工程外系统临时目录;尚未在真实 Windows、GNOME/SSH 或 Pi TUI 进行视觉验收。 +## ANSI 四象限接缝优化(2026-10-30) + +- macOS arm64 的 CLI debug 构建和 `doctor --json` 通过。工程外 ASan/UBSan 检查覆盖低对比合并、横纵细线/斜线/双色图案、等亮度彩色边缘、随机 NV12 输入的失真上限、局部更新范围和重复转换确定性。 +- 合成低对比图案中,3,072 个非空白字形全部合并为纯背景。合成渐变纹理的加权接缝误差由约 207,373 降至 180,187(13.1%);仅统计两端均未合并为纯色的边界时,由约 198,007 降至 169,621。指标衡量相邻样本的重建色差相对源图色差的偏离,并降低真实高对比边缘的权重;不代表实机条纹减少的百分比。 +- 8/24/256 色输出、half/ASCII 与原实现一致性、不同视口/比例/缩放的布局和鼠标映射、静止网格差分及 Kitty 输出检查通过。 +- 本机 `-O2`、1920×1080 输入、240×100 字符视口、每项 12 次转换的合成基准:平滑画面约 7.14 → 6.03 ms/帧,随机亮度噪声约 5.44 → 7.73 ms/帧。仅衡量转换,不包含视频解码、ANSI 输出和终端绘制。 +- 额外检查代码、基准和数据均位于工程外系统临时目录;未新增工程内测试,未连接真实远端或进行终端字形/行距的视觉验收。 + +## ANSI 转换与输出加速(2026-10-30) + +- macOS arm64 CLI debug 构建和 `doctor --json` 通过。工程外 ASan/UBSan 与 `-O2` 检查均通过:加速后的转换与此前接缝优化版本逐格颜色/字形一致,覆盖随机 NV12、低对比及平滑纹理、half/quadrant/ASCII、缩放和细节模式;Kitty 数据保持一致。 +- ANSI 转义序列模拟器验证最终屏幕一致,覆盖全部四象限字形、8/24/256 色、纯背景及色板复用、首尾单元、局部修改、视口/偏移变化和可独立绘制的 bridge 字符行。未向工程新增测试文件或用例。 +- 本机 `-O2`、1920×1080 输入、240×100 字符视口、每项 24 次转换:平滑画面约 6.26 → 4.83 ms/帧,随机亮度噪声约 7.75 → 5.72 ms/帧。保留面积滤波、低对比合并及边界连续性约束。 +- 240×100 真彩色合成网格、每项 100 次输出生成:整帧约 2.03 → 1.43 ms;120×50 单元的局部修改约 1.02 → 0.36 ms,输出 461,743 → 231,026 字节;2×3 单元的光标修改约 0.072 → 0.013 ms,输出 27,763 → 262 字节。纯背景色块整帧输出 875,352 → 449,826 字节。 +- 这些结果衡量转换、编码及输出量,不包含实际终端绘制或网络传输。整帧高色彩变化图案仍需约 923 KB,仍受 1 MiB/s 输出预算限制;不能将转换耗时的倒数作为远程显示帧率。默认本地目标为 8 FPS,`--fps 15` 可请求更高目标,预览订阅还需要对端支持 15 FPS。 +- 额外检查代码、模拟器、基准和产物均位于工程外临时目录;未进行跨平台终端或真实远端帧率验收。 + +## Kitty 帧间黑屏与本地吞吐修复(2026-10-30) + +- macOS arm64 debug、release 构建通过,release `doctor --json` 通过。release 使用工程外独立配置构建,保留工程原有 debug 配置;额外检查脚本、伪终端夹具和数据均位于系统临时目录。 +- ASan/UBSan 检查及 Kitty 协议模拟验证:ID 31/32 交替上传,所有上传分块完成前旧图保持显示;先创建新图 placement,再释放旧图;最多两个完整图像驻留,切换后只保留一个。新旧输出的解压 RGB 和尺寸逐字节一致,覆盖尺寸、偏移和缩放变化;单帧小于 2MiB,全部分块边界的中断清理通过。 +- ANSI 的 half/quadrant/ASCII 和 8/24/256 色共 12 种组合,转换单元及输出字节与此次修改前一致。 +- 使用生产 `View`、`Output` 和 Kitty 编码代码、`-O2`、合成 NV12 持续回调的本地 PTY 对比:目标 8 FPS,单帧约 1.25 MB,稳定完整接收速率约 0.84 → 6.71 FPS,帧间隔中位数约 1,192 → 148 ms。只衡量完整输出接收,不包含真实网络、视频解码和 Ghostty GPU 显示,不能作为实机帧率承诺。 +- 本地 Kitty 的日志预算为 8MiB/s;分别仅设置 `SSH_TTY`、`SSH_CONNECTION`、`SSH_CLIENT` 时均保持 1MiB/s,ANSI 也保持 1MiB/s。连续帧 ID 仅在完整写入后交替,退出清理两个 ID。 +- 慢速 PTY 在第二帧上传中退出:首帧保留、第二帧未提交,约 2.06s 后退出;两个 ID 清理、备用屏幕/光标控制序列和原 termios 均恢复。测试发现原单次清理写入会在满缓冲区时丢失,现增加至多 500ms 的非阻塞短写重试。 +- 模拟父 shell 保持会话、终端完全停止读取的 PTY:CLI 约 2.53s 后返回,termios 和非阻塞标志恢复。POSIX 恢复改为 `TCSANOW` 加输入清理,避免 `TCSAFLUSH` 等待输出排空。完全不读取时,屏幕清理字节不保证送达。 +- 本机安装 Ghostty 1.4.0,但自动化工具拒绝访问其窗口,未完成 Ghostty 实窗视觉验收。npm 安装的 CLI 不会随本地构建自动更新,复测需要直接运行本次 release 产物。 + +## 本地 Kitty 共享内存与 30 FPS 目标(2026-10-30) + +- 本地 POSIX Kitty 增加共享内存能力探测、唯一名称、0600 权限、上传确认和 500ms 超时回退。单个待确认共享缓冲区不超过 1920×1080×3 字节;确认前保留旧画面,退出释放未确认对象。Windows 暂保留内联路径,SSH 不发共享内存探测。 +- 共享内存可用且未显式指定 `--fps` 时默认目标 30;自动回退内联恢复 8,显式帧率保持不变。目标大于 15 使用现有预览协议的交互模式 `fps=60, frame_rate=30`;没有新增协议取值或修改被控端。 +- 工程外 ASan/UBSan 检查通过:110 组随机帧/视口/宽高比/缩放/细节模式,整数面积滤波相对原浮点参考的每通道最大误差为 1/255;内联 Kitty 和 ANSI 保持原输出。覆盖共享内存权限、读取内容、命名、容量限制、双方 unlink、分片确认及非法/溢出/重复 ID 回应。 +- 使用实际 release CLI 对象代码、1920×1080 合成噪声 NV12 连续回调、1568×1088 报告视口的 PTY 测试:内联完整接收约 4.06 FPS,共享内存约 29.04 FPS;平均本地处理约 166.6 → 12.3 ms,终端字节流约 1,703,174 → 188 字节/帧。共享内存仍传输完整 RGB,并非总数据量只有 188 字节;数据复制、终端绘制和实际网络另有成本。 +- 显式 8 FPS、探测不支持/静默、第三帧上传报错/丢失确认、80ms 延迟确认、SSH 禁用探测均通过。回退前不显示未确认帧,运行时回退仅触发一次,并正确恢复自动目标和协议请求;所有共享对象均清理。第二帧上传后立即退出时,首帧仍是已提交画面,未确认对象及两个图像 ID 均清理。 +- 相同合成输入的 debug 构建约 5.91 FPS,本地处理约 163 ms,说明 `-O0` 仍明显影响像素转换;性能验收使用 release。上述精确数字是本地测试接收速率,不是 Ghostty GPU 显示帧率。 +- 用户在本机 Ghostty 运行新版 release 后反馈“明显流畅了”。额外测试源码、数据、构建夹具和产物均位于系统临时目录,未在工程或子模块新增测试。 +- 该次实机日志确认 `kitty_transport=shared_memory`、目标 30、预览请求已接受、运行时回退 0 次。会话约 13.61s 完整提交 273 帧,平均提交约 20.06 FPS(包含约 1.49s 首帧启动等待);平均本地处理约 10.87ms、共享内存确认约 0.42ms。该日志记录提交速率,不测量终端 GPU 呈现频率。 + ## 实测边界 本地检查不涵盖 Linux / Windows 完整构建、跨平台远程控制、Kitty/SSH 实际终端显示、真实 Pi TUI 或长时间资源占用。五个平台 CI 配置与 npm 包名均不代表已发布或已验收。 diff --git a/apps/cli/main.cpp b/apps/cli/main.cpp index f1777214..2db91182 100644 --- a/apps/cli/main.cpp +++ b/apps/cli/main.cpp @@ -31,6 +31,11 @@ public: explicit Output(int fd) : fd_(fd), nonblocking_(fd, true), last_(Clock::now()) {} bool Idle() const { return pending_.empty(); } + size_t ByteRate() const { return byte_rate_; } + void SetByteRate(size_t rate) { + byte_rate_ = rate; + last_ = Clock::now(); + } void Put(std::string text) { if (!Idle() || text.size() > 2 * 1024 * 1024) throw std::runtime_error("output budget exceeded"); @@ -41,7 +46,7 @@ public: auto now = Clock::now(); credit_ = std::min( 1048576.0, - credit_ + std::chrono::duration(now - last_).count() * 1048576); + credit_ + std::chrono::duration(now - last_).count() * byte_rate_); last_ = now; if (Idle()) return false; @@ -75,6 +80,7 @@ private: NonBlocking nonblocking_; std::string pending_; size_t offset_ = 0; + size_t byte_rate_ = 1048576; double credit_ = 1048576; Clock::time_point last_; }; @@ -93,7 +99,24 @@ struct Tty { "\033[0m\033[?1003l\033[?1002l\033[?1000l" "\033[?1006l\033[?1004l\033[?2004l\033[" "?7h\033[?25h\033[?1049l"; - Write(out, end.data(), end.size()); + // A full PTY can reject even this small cleanup after an interrupted image. + // Retry short writes without indefinitely blocking exit on a stalled reader. + try { + NonBlocking nonblocking(out, true); + const auto until = Clock::now() + std::chrono::milliseconds(500); + size_t offset = 0; + while (offset < end.size() && Clock::now() < until) { + auto n = Write(out, end.data() + offset, end.size() - offset); + if (n > 0) + offset += n; + else if (n < 0 && errno != EAGAIN && errno != EINTR) + break; + else + WaitIo(-1, out, 10); + } + } catch (...) { + // A closed terminal must not prevent RawTerminal from restoring modes. + } } }; Viewport Size(int *status_rows = nullptr, @@ -242,10 +265,11 @@ std::string WaitingScreen(const Viewport &v, const std::string &title, } int View(SessionOptions options, const std::string &device, const std::string &renderer, bool control, int fd, AnsiDetail detail, - double zoom, bool native, double cell_aspect) { + double zoom, bool native, double cell_aspect, bool fps_explicit) { if (!IsTty(0) || !IsTty(fd) || Env("TERM") == "dumb") - throw std::runtime_error("connecting needs an interactive ANSI terminal; use " - "bridge --stdio for a host"); + throw std::runtime_error( + "connecting needs an interactive ANSI terminal; use " + "bridge --stdio for a host"); auto password = Password(); if (interrupted) return 130; @@ -269,6 +293,42 @@ int View(SessionOptions options, const std::string &device, if (renderer == "kitty" && !kitty) throw std::runtime_error( "Kitty probe failed or multiplexer active; use --renderer ansi"); + // Local pixel frames need more throughput than character output. Keep the + // conservative budget for SSH and the existing one-message/burst limits. + const bool local_terminal = Env("SSH_CONNECTION").empty() && + Env("SSH_CLIENT").empty() && + Env("SSH_TTY").empty(); + if (kitty && local_terminal) + output.SetByteRate(8 * 1024 * 1024); + SharedImage shared_image; + bool shared_enabled = false; + const uint8_t probe_pixel[] = {0, 0, 0}; + if (kitty && local_terminal && output.Idle() && + shared_image.Assign(probe_pixel, sizeof(probe_pixel))) { + output.Put(KittySharedUpload(shared_image.Name(), 1, 1, false, true)); + std::string reply; + const auto until = Clock::now() + std::chrono::milliseconds(250); + while (Clock::now() < until && !interrupted) { + output.Pump(); + WaitReadable(0, 10); + char bytes[1024]; + const auto n = Read(0, bytes, sizeof(bytes)); + if (n > 0 && reply.size() < 8192) + reply.append(bytes, n); + const auto pos = reply.find("\033_Gi=33;"); + const auto end = + pos == std::string::npos ? pos : reply.find("\033\\", pos); + if (end != std::string::npos) { + const auto ack = ParseKittyAck(reply.substr(pos, end + 2 - pos)); + shared_enabled = ack && ack->ok; + break; + } + } + shared_image.Reset(); + } + bool automatic_fast_fps = shared_enabled && !fps_explicit; + if (automatic_fast_fps) + options.fps = 30; ControllerSession s; Input input; input.control = control; @@ -279,6 +339,11 @@ int View(SessionOptions options, const std::string &device, s.SetControl(control); std::optional displayed, pending; bool pending_frame = false; + bool shared_upload = false; + std::optional shared_deadline; + std::string pending_clear; + uint64_t shared_fallbacks = 0, rendered = 0, shared_acks = 0; + double render_ms = 0, shared_ack_ms = 0; int last_status_rows = 3; Viewport last_view = Size(&last_status_rows, &input.terminal_metrics, cell_aspect); @@ -292,8 +357,24 @@ int View(SessionOptions options, const std::string &device, double queue_ms = 0; auto next = Clock::now(), status_tick = next; auto size_query_tick = next; + const auto disable_shared = [&] { + shared_enabled = shared_upload = false; + shared_deadline.reset(); + shared_image.Reset(); + ++shared_fallbacks; + if (automatic_fast_fps) { + automatic_fast_fps = false; + options.fps = 8; + s.SetPreview(options.fps, options.preview_width, options.preview_height, + options.quality, options.max_bitrate_bps); + } + }; const auto pump_output = [&] { - if (output.Pump() && pending_frame) { + if (!output.Pump()) + return; + if (shared_upload) { + shared_deadline = Clock::now() + std::chrono::milliseconds(500); + } else if (pending_frame) { displayed = std::move(pending); pending.reset(); pending_frame = false; @@ -317,6 +398,8 @@ int View(SessionOptions options, const std::string &device, const auto previous_error = input.error; input.Feed(n > 0 ? std::string(bytes, n) : "", s, displayed ? &*displayed : nullptr); + if (input.exit || interrupted) + break; if (previous_command != input.command_mode || previous_control != input.control || previous_error != input.error) status_tick = Clock::now(); @@ -324,6 +407,35 @@ int View(SessionOptions options, const std::string &device, if (s.Finished()) break; auto now = Clock::now(); + if (shared_upload && shared_deadline) { + const auto ack = input.kitty_ack; + const bool replied = ack && ack->image_id == (submitted % 2 ? 32u : 31u); + if (replied || now >= *shared_deadline) { + if (replied && ack->ok) { + shared_ack_ms += + std::chrono::duration( + now - (*shared_deadline - std::chrono::milliseconds(500))) + .count(); + ++shared_acks; + shared_upload = false; + shared_deadline.reset(); + shared_image.Reset(); + // Display only an acknowledged upload. A missing/failed upload must + // leave the previous image and pointer mapping intact. + output.Put(pending_clear + + KittyPresent(pending->viewport, submitted % 2 != 0)); + pending_frame = true; + waiting_visible = false; + } else { + disable_shared(); + pending.reset(); + redraw = true; + next = now; + } + pending_clear.clear(); + } + } + input.kitty_ack.reset(); int status_rows; auto v = Size(&status_rows, &input.terminal_metrics, cell_aspect); v.detail = detail; @@ -338,7 +450,7 @@ int View(SessionOptions options, const std::string &device, } const bool grid_resized = v.columns != last_view.columns || v.rows != last_view.rows; - if (output.Idle() && + if (output.Idle() && !shared_upload && (grid_resized || v.cell_aspect != last_view.cell_aspect || v.pixel_width != last_view.pixel_width || v.pixel_height != last_view.pixel_height || @@ -354,12 +466,12 @@ int View(SessionOptions options, const std::string &device, } // Query through the terminal byte stream so this also works over SSH. // Never wait for a reply or interrupt a partially written video frame. - if (output.Idle() && now >= size_query_tick) { + if (output.Idle() && !shared_upload && now >= size_query_tick) { input.terminal_metrics.BeginQuery(); output.Put(kTerminalSizeQuery); size_query_tick = now + std::chrono::seconds(2); } - if (output.Idle() && now >= status_tick) { + if (output.Idle() && !shared_upload && now >= status_tick) { auto st = s.Status(); const auto last_state = st["state"].get(); std::string progress = @@ -422,7 +534,7 @@ int View(SessionOptions options, const std::string &device, output.Put(std::move(screen)); status_tick = now + std::chrono::seconds(1); } - if (output.Idle() && now >= next) { + if (output.Idle() && !shared_upload && now >= next) { next = now + std::chrono::milliseconds(1000 / options.fps); bool fresh = false; if (auto frame = s.TakeFrame(last_frame ? &*last_frame : nullptr); @@ -435,6 +547,7 @@ int View(SessionOptions options, const std::string &device, if (last_frame && last_frame->generation != s.Generation()) last_frame.reset(); if (last_frame && (fresh || redraw)) { + const auto render_started = Clock::now(); if (fresh) queue_ms = std::chrono::duration( now - last_frame->queued) @@ -446,12 +559,29 @@ int View(SessionOptions options, const std::string &device, for (int row = 0; row < v.rows; ++row) clear += "\033[" + std::to_string(v.row_offset + row + 1) + ";1H\033[0m\033[2K"; - output.Put(clear + (kitty ? Kitty(*last_frame, v) - : Difference(*pending, displayed))); - waiting_visible = false; + if (shared_enabled) { + const auto pixels = KittyPixels(*last_frame, v); + if (shared_image.Assign(pixels.rgb.data(), pixels.rgb.size())) { + output.Put(KittySharedUpload(shared_image.Name(), pixels.width, + pixels.height, submitted % 2 != 0)); + shared_upload = true; + pending_clear = std::move(clear); + } else { + disable_shared(); + } + } + if (!shared_upload) { + output.Put(clear + (kitty ? Kitty(*last_frame, v, submitted % 2 != 0) + : Difference(*pending, displayed))); + waiting_visible = false; + pending_frame = true; + } + render_ms += std::chrono::duration(Clock::now() - + render_started) + .count(); + ++rendered; redraw = false; - pending_frame = true; - if (output.Idle()) { + if (output.Idle() && !shared_upload) { displayed = std::move(pending); pending.reset(); pending_frame = false; @@ -492,6 +622,15 @@ int View(SessionOptions options, const std::string &device, {"reported", input.terminal_metrics.Known()}, {"manual_aspect", cell_aspect > 0}}}, {"submitted", submitted}, + {"renderer", kitty ? "kitty" : "ansi"}, + {"kitty_transport", !kitty ? "none" + : shared_enabled ? "shared_memory" + : "inline"}, + {"target_fps", options.fps}, + {"frame_processing_ms", rendered ? render_ms / rendered : 0}, + {"shared_ack_ms", shared_acks ? shared_ack_ms / shared_acks : 0}, + {"shared_fallbacks", shared_fallbacks}, + {"output_bytes_per_second", output.ByteRate()}, {"output_bytes", output.bytes}, {"last_queue_ms", queue_ms}, {"elapsed_seconds", @@ -778,7 +917,8 @@ Ctrl+T and Ctrl+C pass through in control mode; another Ctrl+] leaves local mode Default ANSI detail: quadrant. Default quality: high (1920x1080); ultra requests up to 3840x2160. Cell aspect is character width/height: auto detects terminal geometry; use --cell-aspect to calibrate. Display index is zero-based. ANSI cannot reproduce full desktop text at fit scale. ---fps controls local output; negotiated previews use supported 1/8/15 FPS. +--fps controls local output. Local Kitty shared memory defaults to 30; other paths default to 8. +Targets above 15 use the negotiated interactive 30 FPS stream; lower targets use 1/8/15 FPS previews. bridge: private NDJSON v1, credentials on a separate inherited descriptor. No IP direct connection, audio playback, clipboard or file transfer support. )"; @@ -829,7 +969,7 @@ int main(int argc, char **argv) { std::string renderer = "auto", device; AnsiDetail detail = AnsiDetail::Quadrant; double zoom = 1, cell_aspect = 0; - bool control = false, stdio = false, native = false; + bool control = false, stdio = false, native = false, fps_explicit = false; int secret_fd = 3; if (command == "view") { if (i >= argc) @@ -860,9 +1000,10 @@ int main(int argc, char **argv) { options.host = value; else if (a == "--port") options.port = number(value, 1, 65535); - else if (a == "--fps" && command == "view") + else if (a == "--fps" && command == "view") { options.fps = number(value, 1, 30); - else if (a == "--display" && command == "view") + fps_explicit = true; + } else if (a == "--display" && command == "view") options.display = number(value, 0, 63); else if (a == "--renderer" && command == "view") renderer = value; @@ -916,7 +1057,7 @@ int main(int argc, char **argv) { try { return command == "view" ? View(options, device, renderer, control, output.value, detail, zoom, native, - cell_aspect) + cell_aspect, fps_explicit) : Bridge(output.value, options, secret_fd); } catch (const std::exception &e) { if (command == "view") { diff --git a/apps/cli/terminal_io.h b/apps/cli/terminal_io.h index 60a89da7..7e5ca1e5 100644 --- a/apps/cli/terminal_io.h +++ b/apps/cli/terminal_io.h @@ -10,6 +10,7 @@ #include #include #include +#include #include #include @@ -34,6 +35,21 @@ struct TerminalSize { }; TerminalSize GetTerminalSize(int fd); +// One private, uniquely named pixel buffer. Keep it alive until the terminal +// acknowledges reading it; Reset also unlinks objects the terminal left behind. +class SharedImage { +public: + SharedImage(); + ~SharedImage(); + bool Assign(const uint8_t *data, size_t size); + void Reset(); + const std::string &Name() const; + +private: + struct State; + std::unique_ptr state_; +}; + class NonBlocking { public: explicit NonBlocking(int fd, bool output = false); diff --git a/apps/cli/terminal_io_posix.cpp b/apps/cli/terminal_io_posix.cpp index cdeb1ab1..e2b21585 100644 --- a/apps/cli/terminal_io_posix.cpp +++ b/apps/cli/terminal_io_posix.cpp @@ -1,10 +1,12 @@ #ifndef _WIN32 #include "terminal_io.h" #include +#include #include #include #include #include +#include #include #include #include @@ -78,6 +80,66 @@ TerminalSize GetTerminalSize(int fd) { } return size; } +struct SharedImage::State { + std::string name; + void *data = MAP_FAILED; + size_t size = 0; +}; +SharedImage::SharedImage() : state_(std::make_unique()) {} +SharedImage::~SharedImage() { Reset(); } +const std::string &SharedImage::Name() const { return state_->name; } +void SharedImage::Reset() { + if (state_->data != MAP_FAILED) + munmap(state_->data, state_->size); + if (!state_->name.empty()) + shm_unlink(state_->name.c_str()); + state_->data = MAP_FAILED; + state_->size = 0; + state_->name.clear(); +} +bool SharedImage::Assign(const uint8_t *data, size_t size) { + Reset(); + if (!data || !size || size > 1920 * 1080 * 3) + return false; + static std::atomic sequence{0}; + int fd = -1; + for (int attempt = 0; attempt < 16; ++attempt) { + // Fits Darwin's 31-byte name limit. O_EXCL prevents opening another + // process's object, including a stale name from a recycled process ID. + auto name = "/cdx-" + std::to_string(getpid()) + "-" + + std::to_string(++sequence); + fd = shm_open(name.c_str(), O_RDWR | O_CREAT | O_EXCL, 0600); + if (fd >= 0) { + state_->name = std::move(name); + break; + } + if (errno != EEXIST) + return false; + } + if (fd < 0) + return false; + fcntl(fd, F_SETFD, FD_CLOEXEC); +#ifdef __linux__ + // Reserve tmpfs space before touching the mapping; a successful ftruncate + // alone can still SIGBUS later when /dev/shm is full. + if (posix_fallocate(fd, 0, size) != 0) { + close(fd); + Reset(); + return false; + } +#endif + if (ftruncate(fd, size) == 0) { + state_->data = mmap(nullptr, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); + state_->size = size; + } + close(fd); + if (state_->data == MAP_FAILED) { + Reset(); + return false; + } + std::memcpy(state_->data, data, size); + return true; +} struct NonBlocking::State { int fd, flags; }; @@ -107,7 +169,11 @@ RawTerminal::RawTerminal(int input, int) : state_(std::make_unique()) { } } RawTerminal::~RawTerminal() { - tcsetattr(state_->fd, TCSAFLUSH, &state_->saved); + // TCSAFLUSH waits for queued output, which can hang exit when the terminal + // stops consuming a partially written image. Restore immediately and flush + // only pending input so protocol replies do not reach the next shell prompt. + tcsetattr(state_->fd, TCSANOW, &state_->saved); + tcflush(state_->fd, TCIFLUSH); fcntl(state_->fd, F_SETFL, state_->flags); } std::string Password() { diff --git a/apps/cli/terminal_io_windows.cpp b/apps/cli/terminal_io_windows.cpp index cc0b9142..bce36425 100644 --- a/apps/cli/terminal_io_windows.cpp +++ b/apps/cli/terminal_io_windows.cpp @@ -330,6 +330,15 @@ TerminalSize GetTerminalSize(int) { CloseHandle(handle); return size; } +struct SharedImage::State { + std::string name; +}; +SharedImage::SharedImage() : state_(std::make_unique()) {} +SharedImage::~SharedImage() = default; +bool SharedImage::Assign(const uint8_t *, size_t) { return false; } +void SharedImage::Reset() {} +const std::string &SharedImage::Name() const { return state_->name; } + struct NonBlocking::State { HANDLE handle; DWORD mode = 0; diff --git a/apps/cli/xmake.lua b/apps/cli/xmake.lua index e980d428..16c410cc 100644 --- a/apps/cli/xmake.lua +++ b/apps/cli/xmake.lua @@ -44,5 +44,7 @@ target("crossdesk-cli") if is_plat("windows") then add_syslinks("advapi32", "user32") add_defines("NOMINMAX", "WIN32_LEAN_AND_MEAN") + elseif is_plat("linux") then + add_syslinks("rt") end target_end() diff --git a/libs/controller/src/controller_session.cpp b/libs/controller/src/controller_session.cpp index ace23e73..7473351b 100644 --- a/libs/controller/src/controller_session.cpp +++ b/libs/controller/src/controller_session.cpp @@ -335,9 +335,13 @@ void ControllerSession::Preview() { s.request_id = ++preview_request_; preview_accepted_ = false; s.fps = options_.fps == 0 ? 0 + : options_.fps > 15 ? 60 : options_.fps <= 1 ? 1 : options_.fps <= 8 || !fps15_ ? 8 : 15; + // 60 is the existing interactive-mode sentinel, not a 60 FPS request. + // The encoder cadence is independently specified by frame_rate. + s.frame_rate = 30; s.audio = false; s.quality = options_.quality; if (resolution_) { diff --git a/libs/terminal/include/terminal_renderer.h b/libs/terminal/include/terminal_renderer.h index 2229952b..9e34e3c4 100644 --- a/libs/terminal/include/terminal_renderer.h +++ b/libs/terminal/include/terminal_renderer.h @@ -63,10 +63,27 @@ std::optional> MapPointer(const Grid &, int column, int row); std::vector Lines(const Grid &); std::string Difference(const Grid &, const std::optional &previous); -std::string Kitty(const DecodedFrame &, const Viewport &); +struct PixelFrame { + int width = 0, height = 0; + std::vector rgb; +}; +PixelFrame KittyPixels(const DecodedFrame &, const Viewport &); +std::string KittySharedUpload(const std::string &name, int width, int height, + bool alternate_buffer, bool query = false); +std::string KittyPresent(const Viewport &, bool alternate_buffer); +struct KittyAck { + uint32_t image_id = 0; + bool ok = false; +}; +std::optional ParseKittyAck(const std::string &); +// Toggle alternate_buffer only after the previous output has been written in +// full. Upload to the other image ID before replacing the visible placement. +std::string Kitty(const DecodedFrame &, const Viewport &, + bool alternate_buffer = false); inline constexpr char kKittyQuery[] = "\033_Gi=31,s=1,v=1,a=q,t=d,f=24;AAAA\033\\"; -inline constexpr char kKittyDelete[] = "\033_Ga=d,d=I,i=31,q=2\033\\"; +inline constexpr char kKittyDelete[] = + "\033_Ga=d,d=I,i=31,q=2\033\\\033_Ga=d,d=I,i=32,q=2\033\\"; bool KittyReply(const std::string &response); struct StatusBar { std::string progress, compact_progress, error, view, network; @@ -89,6 +106,7 @@ public: uint64_t mouse_legacy_reports = 0, mouse_forwarded = 0; uint64_t mouse_send_failures = 0; TerminalMetrics terminal_metrics; + std::optional kitty_ack; private: std::string pending_; diff --git a/libs/terminal/src/input.cpp b/libs/terminal/src/input.cpp index 4f89e1ea..cd21621f 100644 --- a/libs/terminal/src/input.cpp +++ b/libs/terminal/src/input.cpp @@ -209,6 +209,8 @@ void Input::Feed(const std::string &bytes, ControllerSession &s, auto end = pending_.find("\033\\", 2); if (end == std::string::npos) return; + if (auto ack = ParseKittyAck(pending_.substr(0, end + 2))) + kitty_ack = ack; pending_.erase(0, end + 2); continue; } diff --git a/libs/terminal/src/terminal_renderer.cpp b/libs/terminal/src/terminal_renderer.cpp index d7d38a72..b21ca0f2 100644 --- a/libs/terminal/src/terminal_renderer.cpp +++ b/libs/terminal/src/terminal_renderer.cpp @@ -1,6 +1,7 @@ #include "terminal_renderer.h" #include #include +#include #include #include #include @@ -88,52 +89,163 @@ static uint32_t Sample(const DecodedFrame &f, double x0, double y0, double x1, }; return (round(0) << 16) | (round(1) << 8) | round(2); } -static Cell Quadrant(const std::array &pixels) { - // Enumerate all seven two-colour partitions (complements are equivalent). - static constexpr uint32_t glyphs[16] = { - 0x20, 0x2598, 0x259D, 0x2580, 0x2596, 0x258C, 0x259E, 0x259B, - 0x2597, 0x259A, 0x2590, 0x259C, 0x2584, 0x2599, 0x259F, 0x2588}; - Cell best{}; - double best_error = 1e30; - for (unsigned mask = 1; mask < 8; ++mask) { - double sums[2][3] = {}; - unsigned count[2] = {}; +using QuadrantPixels = std::array; +struct QuadrantFit { + Cell cell; + QuadrantPixels pixels{}; + double error = 0; +}; +struct QuadrantEdge { + double weight = 0; + std::array error{}; +}; +using QuadrantEdges = std::array; +static int Channel(uint32_t color, int channel) { + return (color >> (16 - 8 * channel)) & 255; +} +// Seven partitions suffice; their complements produce the same pixel colours. +static constexpr uint32_t kQuadrantGlyphs[] = {0x20, 0x2598, 0x259D, 0x2580, + 0x2596, 0x258C, 0x259E, 0x259B}; +static QuadrantFit Quadrant(const QuadrantPixels &pixels) { + int channels[4][3], total[3] = {}, squares = 0; + uint32_t mean = 0; + bool flat = true; + for (int c = 0; c < 3; ++c) { + int lo = 255, hi = 0; for (int i = 0; i < 4; ++i) { - int g = (mask >> i) & 1; - ++count[g]; - sums[g][0] += pixels[i] >> 16; - sums[g][1] += (pixels[i] >> 8) & 255; - sums[g][2] += pixels[i] & 255; + int value = channels[i][c] = Channel(pixels[i], c); + lo = std::min(lo, value); + hi = std::max(hi, value); + total[c] += value; + squares += value * value; + } + // Keep the per-channel threshold small to retain faint/coloured detail + // above this range, rather than judging flatness by average luminance. + flat = flat && hi - lo <= 6; + mean |= uint32_t((total[c] + 2) / 4) << (16 - 8 * c); + } + if (flat) { + QuadrantFit fit{ + {mean, mean, 0x20}, {mean, mean, mean, mean}, double(squares)}; + for (int c = 0; c < 3; ++c) { + int value = Channel(mean, c); + fit.error += 4 * value * value - 2 * value * total[c]; + } + return fit; + } + // Reuse subset sums. Integer rounded means and sum-of-squares give exactly + // the original RGB squared error without rescanning four pixels per mask. + static constexpr int counts[] = {0, 1, 1, 2, 1, 2, 2, 3}; + int sums[8][3] = {}; + QuadrantFit best; + int best_error = INT32_MAX, best_mask = 0; + for (int mask = 1; mask < 8; ++mask) { + int bit = mask & 1 ? 0 : mask & 2 ? 1 : 2; + int rest = mask & (mask - 1), count = counts[mask]; + uint32_t fg = 0, bg = 0; + int error = squares; + for (int c = 0; c < 3; ++c) { + int sum = sums[mask][c] = sums[rest][c] + channels[bit][c]; + int front = (sum + count / 2) / count; + int back = (total[c] - sum + (4 - count) / 2) / (4 - count); + fg |= uint32_t(front) << (16 - 8 * c); + bg |= uint32_t(back) << (16 - 8 * c); + error += count * front * front - 2 * front * sum + + (4 - count) * back * back - 2 * back * (total[c] - sum); } - uint32_t colors[2] = {}; - double error = 0; - for (int g = 0; g < 2; ++g) - for (int c = 0; c < 3; ++c) { - sums[g][c] = std::round(sums[g][c] / count[g]); - colors[g] |= uint32_t(sums[g][c]) << (16 - 8 * c); - } - for (int i = 0; i < 4; ++i) - for (int c = 0; c < 3; ++c) { - double d = double((pixels[i] >> (16 - 8 * c)) & 255) - - sums[(mask >> i) & 1][c]; - error += d * d; - } if (error < best_error) { best_error = error; - best = {colors[1], colors[0], glyphs[mask]}; + best_mask = mask; + best.cell = {fg, bg, kQuadrantGlyphs[mask]}; } } - if (best.top == best.bottom) - best.glyph = 0x20; + best.error = best_error; + for (int i = 0; i < 4; ++i) + best.pixels[i] = (best_mask >> i) & 1 ? best.cell.top : best.cell.bottom; + if (best.cell.top == best.cell.bottom) + best.cell.glyph = 0x20; return best; } -static std::string EncodeUtf8(uint32_t c) { - if (c < 128) - return std::string(1, char(c)); - if (c < 2048) - return {char(0xc0 | (c >> 6)), char(0x80 | (c & 63))}; - return {char(0xe0 | (c >> 12)), char(0x80 | ((c >> 6) & 63)), - char(0x80 | (c & 63))}; +static QuadrantFit RefineQuadrant(const QuadrantPixels &pixels, + const QuadrantEdges &edges, + const QuadrantFit &initial) { + int channels[4][3]; + double weighted[4][3]; + for (int i = 0; i < 4; ++i) + for (int c = 0; c < 3; ++c) { + channels[i][c] = Channel(pixels[i], c); + weighted[i][c] = + (1 + edges[i].weight) * channels[i][c] + edges[i].error[c]; + } + auto score = [&](const QuadrantFit &fit) { + double value = fit.error; + for (int i = 0; i < 4; ++i) + for (int c = 0; c < 3; ++c) { + double d = Channel(fit.pixels[i], c) - channels[i][c]; + // Match neighbouring quantization errors, preserving the source + // gradient. The neighbour-only constant does not affect selection. + value += edges[i].weight * d * d - 2 * edges[i].error[c] * d; + } + return value; + }; + QuadrantFit best = initial; + double best_score = score(best); + const double max_error = initial.error + std::min(48.0, initial.error * .05); + for (unsigned mask = 1; mask < 8; ++mask) { + double sums[2][3] = {}; + double weights[2] = {}; + for (int i = 0; i < 4; ++i) { + int g = (mask >> i) & 1; + double weight = 1 + edges[i].weight; + weights[g] += weight; + for (int c = 0; c < 3; ++c) + sums[g][c] += weighted[i][c]; + } + uint32_t colors[2] = {}; + for (int g = 0; g < 2; ++g) + for (int c = 0; c < 3; ++c) { + sums[g][c] = + std::clamp(std::round(sums[g][c] / weights[g]), 0.0, 255.0); + colors[g] |= uint32_t(sums[g][c]) << (16 - 8 * c); + } + QuadrantFit fit{{colors[1], colors[0], kQuadrantGlyphs[mask]}}; + for (int i = 0; i < 4; ++i) { + fit.pixels[i] = colors[(mask >> i) & 1]; + for (int c = 0; c < 3; ++c) { + double d = channels[i][c] - Channel(fit.pixels[i], c); + fit.error += d * d; + } + } + // Limit the cost of continuity to 5% of the independent fit's error, + // capped at 2 RGB levels RMS over the cell's twelve channel samples. + if (fit.error > max_error) + continue; + double value = score(fit); + if (value < best_score) { + best_score = value; + best = fit; + } + } + if (best.cell.top == best.cell.bottom) + best.cell.glyph = 0x20; + return best; +} +static void AppendUtf8(std::string &out, uint32_t c) { + if (c < 128) { + out += char(c); + } else if (c < 2048) { + out += char(0xc0 | (c >> 6)); + out += char(0x80 | (c & 63)); + } else { + out += char(0xe0 | (c >> 12)); + out += char(0x80 | ((c >> 6) & 63)); + out += char(0x80 | (c & 63)); + } +} +static void AppendNumber(std::string &out, uint32_t value) { + char digits[10]; + auto result = std::to_chars(digits, digits + sizeof(digits), value); + out.append(digits, result.ptr); } Grid Layout(const DecodedFrame &f, Viewport v) { Validate(f, v); @@ -156,6 +268,8 @@ Grid Convert(const DecodedFrame &f, Viewport v) { const auto crop = g.source; const int sx = g.samples_x, sy = g.samples_y; g.cells.resize(size_t(v.columns) * v.rows); + std::vector samples(sx == 2 ? g.cells.size() : 0); + std::vector fits(samples.size()); auto pixel = [&](int x, int y) { const auto &r = g.content; if (x < r.x || y < r.y || x >= r.x + r.width || y >= r.y + r.height) @@ -167,15 +281,65 @@ Grid Convert(const DecodedFrame &f, Viewport v) { }; for (int y = 0; y < v.rows; ++y) for (int x = 0; x < v.columns; ++x) { - auto &c = g.cells[size_t(y) * v.columns + x]; - if (sx == 2) - c = Quadrant({pixel(x * 2, y * 2), pixel(x * 2 + 1, y * 2), - pixel(x * 2, y * 2 + 1), pixel(x * 2 + 1, y * 2 + 1)}); - else { + const auto index = size_t(y) * v.columns + x; + auto &c = g.cells[index]; + if (sx == 2) { + samples[index] = {pixel(x * 2, y * 2), pixel(x * 2 + 1, y * 2), + pixel(x * 2, y * 2 + 1), pixel(x * 2 + 1, y * 2 + 1)}; + fits[index] = Quadrant(samples[index]); + c = fits[index].cell; + } else { c.top = pixel(x, y * sy); c.bottom = sy == 1 ? c.top : pixel(x, y * sy + sy - 1); } } + // Read only independent fits so refinement has no left-to-right/top-to-bottom + // feedback, and a changed cell cannot propagate through the entire frame. + if (sx == 2) + for (int y = 0; y < v.rows; ++y) + for (int x = 0; x < v.columns; ++x) { + const auto index = size_t(y) * v.columns + x; + if (fits[index].cell.glyph == 0x20 || fits[index].error == 0) + continue; + QuadrantEdges edges{}; + auto edge = [&](int i, size_t neighbour, int j) { + int contrast = 0; + for (int c = 0; c < 3; ++c) + contrast = + std::max(contrast, std::abs(Channel(samples[index][i], c) - + Channel(samples[neighbour][j], c))); + // Fade out across real edges instead of smoothing text into its + // background. Equal source colours receive the strongest constraint. + double strength = std::max(0.0, 1 - contrast / 24.0); + double weight = .35 * strength * strength; + edges[i].weight += weight; + for (int c = 0; c < 3; ++c) + edges[i].error[c] += + weight * (Channel(fits[neighbour].pixels[j], c) - + Channel(samples[neighbour][j], c)); + }; + if (x > 0) { + edge(0, index - 1, 1); + edge(2, index - 1, 3); + } + if (x + 1 < v.columns) { + edge(1, index + 1, 0); + edge(3, index + 1, 2); + } + if (y > 0) { + edge(0, index - v.columns, 2); + edge(1, index - v.columns, 3); + } + if (y + 1 < v.rows) { + edge(2, index + v.columns, 0); + edge(3, index + v.columns, 1); + } + if (edges[0].weight + edges[1].weight + edges[2].weight + + edges[3].weight > + 0) + g.cells[index] = + RefineQuadrant(samples[index], edges, fits[index]).cell; + } return g; } std::optional> MapPointer(const Grid &g, int x, int y) { @@ -194,12 +358,10 @@ std::optional> MapPointer(const Grid &g, int x, int y) { (g.source.y + fy * std::max(0, g.source.height - 1)) / std::max(1u, g.source_height - 1)}; } -static std::string Color(uint32_t c, bool fg, int depth) { +static uint32_t TerminalColor(uint32_t c, int depth) { const int r = c >> 16, g = (c >> 8) & 255, b = c & 255; - const std::string prefix = fg ? "\033[38;" : "\033[48;"; if (depth == 24) - return prefix + "2;" + std::to_string(r) + ";" + std::to_string(g) + ";" + - std::to_string(b) + "m"; + return c; if (depth == 256) { constexpr int levels[] = {0, 95, 135, 175, 215, 255}; auto index = [&](int value) { @@ -218,55 +380,101 @@ static std::string Color(uint32_t c, bool fg, int depth) { gv = 8 + 10 * gray; int gray_error = (r - gv) * (r - gv) + (g - gv) * (g - gv) + (b - gv) * (b - gv); - return prefix + "5;" + - std::to_string(gray_error < cube ? 232 + gray - : 16 + 36 * ri + 6 * gi + bi) + - "m"; + return gray_error < cube ? 232 + gray : 16 + 36 * ri + 6 * gi + bi; } - int i = (r >= 128 ? 1 : 0) + (g >= 128 ? 2 : 0) + (b >= 128 ? 4 : 0); - return "\033[" + std::to_string((fg ? 30 : 40) + i) + "m"; + return (r >= 128 ? 1 : 0) + (g >= 128 ? 2 : 0) + (b >= 128 ? 4 : 0); } -static std::string Line(const Grid &g, int y) { - std::string line; - uint32_t fg = UINT32_MAX, bg = UINT32_MAX; - for (int x = 0; x < g.viewport.columns; ++x) { - const auto &c = g.cells[size_t(y) * g.viewport.columns + x]; - if (g.viewport.utf8 && fg != c.top) { - line += Color(c.top, true, g.viewport.colors); - fg = c.top; +static void AppendColor(std::string &out, uint32_t c, bool fg, int depth) { + if (depth == 24 || depth == 256) { + out += fg ? "\033[38;" : "\033[48;"; + if (depth == 24) { + out += "2;"; + AppendNumber(out, c >> 16); + out += ';'; + AppendNumber(out, (c >> 8) & 255); + out += ';'; + AppendNumber(out, c & 255); + } else { + out += "5;"; + AppendNumber(out, c); } - const auto bottom = g.viewport.utf8 ? c.bottom : c.top; - if (bg != bottom) { - line += Color(bottom, false, g.viewport.colors); - bg = bottom; - } - line += g.viewport.utf8 ? EncodeUtf8(c.glyph) : " "; + } else { + out += "\033["; + AppendNumber(out, (fg ? 30 : 40) + c); + } + out += 'm'; +} +struct AnsiColors { + uint32_t fg = UINT32_MAX, bg = UINT32_MAX; +}; +static void AppendCells(std::string &out, const Grid &g, int y, int begin, + int end, AnsiColors &colors) { + for (int x = begin; x < end; ++x) { + const auto &c = g.cells[size_t(y) * g.viewport.columns + x]; + // Spaces need no foreground SGR. Keep backgrounds for block glyphs too, + // since font rasterization can expose them at the cell's edges. + if (g.viewport.utf8 && c.glyph != 0x20) { + auto fg = TerminalColor(c.top, g.viewport.colors); + if (colors.fg != fg) { + AppendColor(out, fg, true, g.viewport.colors); + colors.fg = fg; + } + } + // Track the emitted palette index so quantized colours do not resend SGR. + auto bg = + TerminalColor(g.viewport.utf8 ? c.bottom : c.top, g.viewport.colors); + if (colors.bg != bg) { + AppendColor(out, bg, false, g.viewport.colors); + colors.bg = bg; + } + AppendUtf8(out, g.viewport.utf8 ? c.glyph : 0x20); } - return line + "\033[0m"; } std::vector Lines(const Grid &g) { std::vector lines; lines.reserve(g.viewport.rows); - for (int y = 0; y < g.viewport.rows; ++y) - lines.push_back(Line(g, y)); + for (int y = 0; y < g.viewport.rows; ++y) { + std::string line; + line.reserve(size_t(g.viewport.columns) * 40 + 4); + AnsiColors colors; + AppendCells(line, g, y, 0, g.viewport.columns, colors); + line += "\033[0m"; + lines.push_back(std::move(line)); + } return lines; } std::string Difference(const Grid &g, const std::optional &previous) { std::string out; + AnsiColors colors; const bool same = previous && previous->viewport.columns == g.viewport.columns && previous->viewport.rows == g.viewport.rows && previous->viewport.row_offset == g.viewport.row_offset && previous->viewport.colors == g.viewport.colors && - previous->viewport.utf8 == g.viewport.utf8; + previous->viewport.utf8 == g.viewport.utf8 && + previous->cells.size() == g.cells.size(); for (int y = 0; y < g.viewport.rows; ++y) { - auto start = g.cells.begin() + size_t(y) * g.viewport.columns; - if (!same || - !std::equal(start, start + g.viewport.columns, - previous->cells.begin() + size_t(y) * g.viewport.columns)) - out += "\033[" + std::to_string(g.viewport.row_offset + y + 1) + - ";1H" + Line(g, y); + int begin = 0, end = g.viewport.columns; + if (same) { + const auto offset = size_t(y) * g.viewport.columns; + while (begin < end && + g.cells[offset + begin] == previous->cells[offset + begin]) + ++begin; + while (end > begin && + g.cells[offset + end - 1] == previous->cells[offset + end - 1]) + --end; + } + if (begin == end) + continue; + out += "\033["; + AppendNumber(out, g.viewport.row_offset + y + 1); + out += ';'; + AppendNumber(out, begin + 1); + out += 'H'; + AppendCells(out, g, y, begin, end, colors); } + if (!out.empty()) + out += "\033[0m"; return out; } static std::string Base64(const std::vector &data) { @@ -287,7 +495,132 @@ static std::string Base64(const std::vector &data) { } return out; } -std::string Kitty(const DecodedFrame &f, const Viewport &requested) { +PixelFrame KittyPixels(const DecodedFrame &f, const Viewport &requested) { + auto v = requested; + v.pixel_mode = true; + Validate(f, v); + PixelFrame pixels; + pixels.width = + std::clamp(v.pixel_width > 0 ? v.pixel_width : v.columns * 10, 2, 1920); + pixels.height = + std::clamp(v.pixel_height > 0 ? v.pixel_height : v.rows * 20, 2, 1080); + pixels.rgb.resize(size_t(pixels.width) * pixels.height * 3, 0); + const auto crop = Crop(f, v); + const auto r = Fit(crop.width, crop.height, pixels.width, pixels.height, + double(v.columns) * v.cell_aspect * pixels.height / + (v.rows * pixels.width)); + if (r.width == crop.width && r.height == crop.height) { + for (int y = 0; y < r.height; ++y) + for (int x = 0; x < r.width; ++x) { + const auto c = Pixel(f, crop.x + x, crop.y + y); + auto *dst = &pixels.rgb[(size_t(y + r.y) * pixels.width + x + r.x) * 3]; + dst[0] = c >> 16; + dst[1] = c >> 8; + dst[2] = c; + } + return pixels; + } + // Separable area integration. Integer overlap weights avoid per-pixel + // floor/ceil and repeated NV12 conversion in overlapping vertical samples. + // Adjacent output rows can share only their boundary source row, so one + // cached horizontal row suffices, even when upscaling. + struct Span { + int first, last, begin, end; + }; + std::vector spans; + for (int x = 0; x < r.width; ++x) { + const int begin = x * crop.width, end = begin + crop.width; + spans.push_back( + {begin / r.width, (end + r.width - 1) / r.width, begin, end}); + } + std::vector source(crop.width), horizontal(size_t(r.width) * 3); + std::vector sums(size_t(r.width) * 3); + const uint64_t area = uint64_t(crop.width) * crop.height; + int cached_y = -1; + for (int y = 0; y < r.height; ++y) { + std::fill(sums.begin(), sums.end(), 0); + const int begin = y * crop.height, end = begin + crop.height; + for (int sy = begin / r.height; sy < (end + r.height - 1) / r.height; + ++sy) { + if (cached_y != sy) { + for (int x = 0; x < crop.width; ++x) + source[x] = Pixel(f, crop.x + x, crop.y + sy); + std::fill(horizontal.begin(), horizontal.end(), 0); + for (int x = 0; x < r.width; ++x) { + const auto &span = spans[x]; + for (int sx = span.first; sx < span.last; ++sx) { + const unsigned weight = std::min(span.end, (sx + 1) * r.width) - + std::max(span.begin, sx * r.width); + const auto c = source[sx]; + horizontal[x * 3] += (c >> 16) * weight; + horizontal[x * 3 + 1] += ((c >> 8) & 255) * weight; + horizontal[x * 3 + 2] += (c & 255) * weight; + } + } + cached_y = sy; + } + const unsigned weight = + std::min(end, (sy + 1) * r.height) - std::max(begin, sy * r.height); + for (size_t i = 0; i < sums.size(); ++i) + sums[i] += uint64_t(horizontal[i]) * weight; + } + auto *dst = &pixels.rgb[(size_t(y + r.y) * pixels.width + r.x) * 3]; + for (size_t i = 0; i < sums.size(); ++i) + dst[i] = (sums[i] + area / 2) / area; + } + return pixels; +} +std::string KittySharedUpload(const std::string &name, int width, int height, + bool alternate_buffer, bool query) { + if (name.empty() || name.size() > 128 || width < 1 || width > 1920 || + height < 1 || height > 1080) + throw std::invalid_argument("invalid shared image"); + return "\033_Ga=" + std::string(query ? "q" : "t") + ",t=s,f=24,q=0,i=" + + std::to_string(query ? 33 + : alternate_buffer ? 32 + : 31) + + ",s=" + std::to_string(width) + ",v=" + std::to_string(height) + + ",S=" + std::to_string(size_t(width) * height * 3) + ";" + + Base64(std::vector(name.begin(), name.end())) + "\033\\"; +} +std::string KittyPresent(const Viewport &v, bool alternate_buffer) { + return "\033[" + std::to_string(v.row_offset + 1) + + ";1H\033_Ga=p,i=" + std::to_string(alternate_buffer ? 32 : 31) + + ",p=1,C=1,q=2,c=" + std::to_string(v.columns) + + ",r=" + std::to_string(v.rows) + + "\033\\\033_Ga=d,d=I,i=" + std::to_string(alternate_buffer ? 31 : 32) + + ",q=2\033\\"; +} +std::optional ParseKittyAck(const std::string &s) { + if (s.size() > 1024 || s.size() < 10 || s.compare(0, 3, "\033_G") != 0 || + s.compare(s.size() - 2, 2, "\033\\") != 0) + return {}; + const auto separator = s.find(';', 3); + if (separator == std::string::npos) + return {}; + uint32_t image_id = 0; + for (size_t pos = 3; pos < separator;) { + const auto end = std::min(s.find(',', pos), separator); + if (s.compare(pos, 2, "i=") == 0) { + if (image_id) + return {}; + const auto parsed = + std::from_chars(s.data() + pos + 2, s.data() + end, image_id); + if (parsed.ec != std::errc{} || parsed.ptr != s.data() + end || !image_id) + return {}; + } else if (s.compare(pos, 2, "p=") == 0 && + s.substr(pos, end - pos) != "p=0") { + return {}; // Only upload/query acknowledgements are relevant here. + } + pos = end + 1; + } + if (!image_id) + return {}; + return KittyAck{image_id, + s.substr(separator + 1, s.size() - separator - 3) == "OK"}; +} +std::string Kitty(const DecodedFrame &f, const Viewport &requested, + bool alternate_buffer) { auto v = requested; v.pixel_mode = true; Validate(f, v); @@ -327,16 +660,22 @@ std::string Kitty(const DecodedFrame &f, const Viewport &requested) { w = std::max(2, w * 3 / 4); h = std::max(2, h * 3 / 4); } - std::string out = "\033[" + std::to_string(v.row_offset + 1) + ";1H"; + const auto image_id = std::to_string(alternate_buffer ? 32 : 31); + std::string out; + out.reserve(b64.size() + (b64.size() / 4096 + 1) * 12 + 256); for (size_t i = 0; i < b64.size(); i += 4096) { out += "\033_G"; if (i == 0) - out += "a=T,t=d,f=24,o=z,q=2,i=31,p=1,C=1,s=" + std::to_string(w) + - ",v=" + std::to_string(h) + ",c=" + std::to_string(v.columns) + - ",r=" + std::to_string(v.rows) + ","; - out += "m=" + std::to_string(i + 4096 < b64.size() ? 1 : 0) + ";" + - b64.substr(i, 4096) + "\033\\"; + out += "a=t,t=d,f=24,o=z,q=2,i=" + image_id + ",s=" + + std::to_string(w) + ",v=" + std::to_string(h) + ","; + out += i + 4096 < b64.size() ? "m=1;" : "m=0;"; + out.append(b64, i, 4096); + out += "\033\\"; } + // Re-transmitting a visible image ID can remove its placement as soon as + // upload starts. Keep the old image throughout chunking/short writes, then + // place the fully uploaded image before freeing the previous one. + out += KittyPresent(v, alternate_buffer); return out; } bool KittyReply(const std::string &s) {