Screen State-Based Automation Execution

A fixed sequence of key presses assumes the application is ready at the same time on every run. This project replaces that assumption with conditions derived from screen content. A sequence can wait for a match, branch when a state is present or keep an input held until a condition becomes true.

The visual editor uses Blockly to represent bindings, triggers and actions. A Python runtime performs native screen capture, hotkey handling and input execution, while pywebview connects that runtime to the editor. A browser fallback supports the editor surface, but the native desktop bridge remains responsible for operating the computer.

Pixel sets record monitor-relative coordinates and expected RGB colours with a tolerance. A full-screen reference can instead provide a template comparison. Both approaches turn visual evidence into a match score, allowing the threshold to be adjusted to the stability of the target interface.

A match fraction is literal: four matching samples out of five give 0.8. It is not a probability that the program understands the scene. Choosing stable pixels and appropriate tolerance matters more than presenting that number as intelligence. Animation, display scaling, window position and changing colours can all affect a match.

The capture preview updates at a modest rate, while the comparison target follows the condition being evaluated at runtime. This keeps inspection connected to the active wait or trigger rather than leaving a stale reference on screen.

Conditions can combine screen or pixel matches with AND, OR and NOT. They can also inspect target state, whether a binding is active or enabled, numeric comparisons and match-score thresholds. Sharing this condition representation across triggers, branches and loops reduces the number of subtly different ways to describe the same rule.

Older trigger fields are migrated into the unified trigger-condition representation. Saving a workspace and loading it again must retain that meaning. JSON round-trip behaviour is therefore part of the editor's correctness, not just a convenience for storing blocks.

The runtime supports presses, waits, timed holds, holds until a condition, loops until a condition, conditional branches and background watchers. These constructs let an automation express both a planned sequence and the checks that decide when it may continue.

Each binding may watch for its trigger, but only one main sequence owns the action-execution slot at a time. A trigger is checked again after that slot is acquired. Without the second check, a binding could queue while a screen matched and execute later after the application had moved somewhere else.

Background watchers belong to the active binding and react to false-to-true transitions. Their work is joined before the main slot is released. That lifecycle prevents a completed sequence from leaving an old watcher competing with the next binding for control.

A visual program needs runtime feedback at the same level as its editing model. Block identifiers travel through execution so the editor can highlight the action or condition currently being processed. Waits remain highlighted for their actual lifetime, while short input actions are still visible long enough to inspect.

Nested execution exposed two different problems. A descendant CSS selector made more than the intended block appear active, and a loop path needed to preserve the child step wrapper so its identifiers reached the feedback channel. Correcting both made the highlight describe real execution rather than merely indicating that some outer block was running.

The editor polls runtime state frequently enough to follow a sequence without treating the UI as a frame-by-frame video stream. The stop hotkey provides an explicit interruption route; visual conditions do not remove the need to supervise automation.

The hardest design issue was coordinating concurrent observation with serial input. Watching several bindings is useful, but allowing each to send keys independently would make the result depend on timing. The shared execution slot and post-acquisition trigger check establish an understandable ordering.

Another issue was keeping the editor, saved document and runtime aligned as the condition model evolved. A block that displays correctly but reloads with a different trigger is a behavioural bug. Migration and round-trip checks protect that contract across all three representations.

State-based automation is more adaptable than a sequence of sleeps, but it still depends on the quality of its observations. This implementation does not infer game state from telemetry or OCR. A visual match remains vulnerable to display and application changes, so thresholds and sample locations must be treated as part of the program.

The broader lesson is that an automation tool has two users: the person defining a sequence and the person trying to understand why it ran. Explicit ownership, visible conditions, persistent block identity and inspectable feedback serve both.