[feat] add batched actions and agent monitoring for MCP

This commit is contained in:
dijunkun committed 2026-11-12 01:43:04 +08:00
1 parent d8739c324d
commit 5047599084
11 files changed
+496 -161

No files matched your search

+6
View File
@@ -120,6 +120,10 @@ All pointer actions and `key_press` accept optional `modifiers`, an array of at
Only one input sequence runs at once. Timed steps are advanced by the owner loop; cancellation, disconnect, display changes and pause release held buttons/keys. A failed send is conservatively considered potentially submitted.
`hello` and status advertise `action_sequence_protocol: "sequence/1"`. An action may be `{type:"sequence", actions:[...], interval_ms:100}` with 1–8 ordinary actions. Nested sequences and `key_down`, `key_up`, `mouse_down`, `mouse_up` are rejected. Sequences require no pre-held inputs, at most 16 KiB of total text and at most 8000 ms of planned input duration including the 0–500 ms inter-action interval. The complete sequence is validated against the initial observation before any input is sent. Only use fixed sequences that require no intermediate visual decision.
Each sub-action finishes its bounded receipt wait before the next starts; errors, acknowledgement timeouts, cancellation, pause and geometry changes stop the remainder. Legacy hosts retain `remote_execution: "unsupported"`; an interval is not an application-readiness guarantee. A sequence has a 25-second overall deadline and one operation-cache entry. Results include `sequence.action_count`, `finished_actions` (finished input stages, not application success) and a zero-based `stopped_at` on failure. `timing_ms.input` measures native input processing including receipt waits and intervals.
New actions require a current frame no older than two seconds as well as a valid observation. Cached operation lookups remain available with a stalled feed or paused control. Failed key/button releases remain tracked for subsequent cleanup, including after control is disabled; a successful local send still does not prove remote execution. Invalid display indices leave existing observations and input sequences intact.
Results use `status: submitted | in_progress | failed_before_submit | outcome_unknown`, with `operation_id`, `retryable`, `side_effect_possible` and, for completed actions, `generation`/`after_frame_id`. The same operation ID and arguments retrieve the cached result; different arguments conflict. Retention is bounded to 128 entries, with entries older than five minutes eligible for removal on a new action. Never retry an uncertain effect with a new ID. `submitted` does not claim remote execution.
@@ -142,6 +146,8 @@ Updated desktop hosts answer the versioned `automation` capability handshake. Wh
`wait_frame` accepts an `observation_id`, `condition` (`changed` or `stable`), `stable_ms` (100–5000, default 500), and `timeout_ms` (0–25000, default 5000). It watches the observation's source region, checks both luma and chroma, and returns `matched`, `wait_timed_out`, `generation`, and `frame_id`. Display changes, disconnection and explicit cancellation terminate pending waits. `crossdesk_wait` exposes this operation through MCP and includes a fresh screenshot when its condition matches.
With `wait_after_frame_supported: true` in `hello`/status, `wait_frame` also accepts `after_frame_id`. Only frames strictly beyond that boundary can match. The MCP action wrapper uses the completed action's frame boundary, runs a bounded visual or UI-condition wait, and returns one final screenshot within the same tool call. A visual match is not a causal acknowledgement; use a relevant cropped observation or a native UI condition for a more specific check. Wait timeouts preserve the input result and never trigger a replay.
## Clipboard
`clipboard` accepts `operation: get | set`. Writes additionally require `operation_id` and `text` (up to 16384 UTF-8 bytes, including an empty string to clear). Reads return up to 128 KiB of text. The existing chunked `web-clipboard` stream carries values and applied/rejected replies. Additive `read`, `empty`, and `clear` frames are enabled by the remote `clipboard` capability. Clipboard access requires control permission and respects human pause. Cancellation of an issued write reports an uncertain effect. Repeated write operation IDs retrieve the cached result.