Google Patents an AI That Reads Your Screen and Clicks Through Apps for You

By Patentlyze Team · Updated Jun 12, 2026

Google is patenting a system that lets an AI look at your screen, number every tappable button and link, then figure out exactly which ones to click — all from a plain-English instruction you type or speak.

How Google's AI sees your screen and takes over

Imagine telling your phone, "Book me the cheapest flight to Chicago for next Friday," and having it actually do it — navigating through an airline app, tapping the right fields, entering dates, and hitting confirm. That's the kind of thing Google's new patent is trying to make real.

The system works by taking a screenshot of whatever app or webpage is on your screen and stamping a little number on every button, link, or input field you could possibly interact with. Then it feeds that labeled image, along with your instruction, into an AI model. The AI reads both, decides which numbered element to interact with and how, and executes the action — no human hand required.

This is less about replacing one app and more about giving AI a universal remote control for any software, whether it was designed for automation or not.

How the numbered-label system guides the AI's clicks

The patent describes a pipeline that connects natural language instructions to real actions inside a graphical user interface (GUI) — basically any app, website, or operating system screen.

Here's the sequence:

• A screenshot of the current screen is captured and processed to identify every interactable element (buttons, text boxes, dropdowns, links).
• Each element gets assigned a numbered label — a technique the patent calls a "set-of-marks" strategy — essentially putting a Post-it note with a number on every clickable thing.
• That annotated image is fed into a vision language model (VLM) — an AI that can simultaneously understand pictures and text — alongside the user's plain-English request.
• The VLM outputs both the action to take (click, type, scroll) and the index number of the exact element to act on.
• The system then executes that action on the live interface.

The numbering scheme is the key innovation here. Rather than asking the AI to pinpoint a pixel coordinate — which is error-prone — it just has to say "click element 7," and the system knows exactly what that maps to on screen. It's a simple trick that dramatically reduces ambiguity for the AI.

What this means for hands-free and automated computing

Most AI assistants today can tell you how to do something on your computer. This system would actually do it for you — inside any app, without that app needing special integration or support. That's a meaningful distinction, because it means the same AI agent could handle a legacy enterprise software tool just as easily as a modern web app.

For everyday users, think of it as a capable assistant who can operate your computer on your behalf when you're overwhelmed, multitasking, or dealing with an interface that's confusing. For businesses, it points toward AI workflows that automate repetitive screen-based tasks — data entry, form submission, report generation — without custom software built for each one.

Editorial commentary on a publicly published patent application. Not legal advice.