GUI Automation
A screen-aware RPA engine that scans a desktop, drafts a step-by-step script, and waits for a human to approve — then runs it, audited keystroke by keystroke.
Client — Mid-market firm running legacy desktop tools with no API
Mocked screen capture, detected UI elements, drafted script, and approval queue — every widget label, keystroke, and coordinate on this page is synthetic.
Most legacy software has no API. A vision-based RPA engine reads the screen the way a person does, drafts a step-by-step script, and waits for a human to approve before anything runs — with a full keystroke audit trail so the regulator can replay the morning.
Scan → Detect → Draft → Approve → Execute
Drafted script
Press Run the pipeline to scan the screen.
Hard gates before execute
- Detection trustall elements above threshold
- Sandbox matchdry-run ↔ live parity
- UI drift guardshape unchanged since draft
- Audit trailkeystroke log enabled
Audit log · replayable
0/0 actionsAwaiting run.
Interactive / built with synthetic data for this page
[ THE CHALLENGE ]
Mid-market operations are held together by desktop apps with no API. Operators click the same ten buttons in the same order every Monday, and any change to the UI breaks the next run. The team needed a screen-aware RPA engine that drafts the click sequence, awaits human approval, and runs it with a keystroke-level audit log.
How we built it
From signal to shipped.
Scan the screen
A vision model trained on legacy desktop UIs detects buttons, fields, and menus — every label rendered on the captured frame. Confidence scores per element drive a deterministic high-trust fast-path and a grounded LLM slow-path for the long tail.
Draft a step-by-step script
Detected elements become a click-target list. An action sequencer drafts a procedural script — coordinates, waits, selectors — grounded in the elements it actually saw. Nothing invented, no synthesised widget the screen never showed.
Approve, then execute
A reviewer approves the script, runs it against a sandbox, watches it click through, then promotes it. Hard gates cover screen-shape drift, dry-run parity, and a quarantine queue for elements below the trust threshold.
Audited end-to-end
Every keystroke and click is logged with the originating screenshot, the matched element, and the script line that drove it. The regulator can replay the morning exactly as it happened.
What shipped.
Element recall ~92% on legacy desktop UIs across the audit corpus.
Two-style authoring: deterministic fast-path + grounded slow-path for the long tail.
Approved scripts run against a sandbox before promotion; quarantine queue for low-trust elements.
Full keystroke + screenshot audit log per run, replayable.
[ NEXT PROJECT ]
FuelCast