Microsoft Files Patent for a Map That Helps AI Click Through Websites
One common way AI assistants use websites is to take a screenshot at every step and ask a big AI model where to click. Microsoft's patent application describes a pre-built map of the site instead, so the assistant already knows where the buttons are.
What Microsoft's website action map does
Imagine telling an AI assistant, renew all my library books for another month, and walking away. An assistant like that can work by taking a screenshot of the web page, sending it to a big AI model, getting told where to click, and repeating that for every single step. The filing says this back-and-forth is slow and costly.
Microsoft's patent application describes a shortcut. A program first crawls a website ahead of time, the way a curious visitor would click around, and writes down a map of what's there: the search bar, the checkout button, what each one does, and how to tell it worked. That map is called an action index.
When you later ask for a task, the AI reads the map once and writes a step-by-step plan. The assistant follows it, checking that each button is on the page before it clicks and that the right page shows up afterward. If something looks off, it asks the AI for help, and it can ask you too.
… identify an actionable GUI element represented in the GUI definition and in the GUI image, the actionable GUI element being configured for receiving an interaction by a user …
Translation: The system finds buttons or links you can click on both the code and the screenshot.
How the index checks each click before and after
The patent has two phases. First, a crawler visits a website or app and collects two things for each page: its GUI definition (the underlying code that describes the page, such as HTML) and an image of how it looks. It gives both to a generative AI model, which picks out groups of related buttons and fields (the filing calls them atoms, such as a search bar) and infers what actions they allow.
For each action, the index records:
- A precondition: a text signature, like the phrase 'Search Products,' whose presence on the page suggests the action will work.
- A postcondition: a signature that appears afterward, like 'Search Results,' confirming it worked.
- For services with an API (a direct, code-level way to talk to a service), the inputs, outputs and how to call it.
The index can be organized as a directed graph (a map where pages are stops and actions are one-way roads between them). The crawler can also run each action to test it.
At run time, an agent asks the AI for a plan built from the index, written in a small custom language. Before each step it checks the precondition in the page code, and after the step it checks the postcondition. If a precondition is missing, it falls back to sending a screenshot to the AI for a replacement action, then updates the index if that works. If that fails too, it asks you.
A presence of the precondition in the GUI indicates a likelihood of success of the action, and a presence of the postcondition in the GUI, after performing the action, indicates success of the action.
Translation: It checks for clues before clicking to see if it will work, and checks again afterward to confirm it succeeded.
Why cheaper web chores matter for AI assistants
If AI assistants are going to handle chores like downloading last year's bank statements or buying hiking boots (both examples in the filing), they have to do it cheaply and quickly or nobody will bother. Skipping the screenshot at every step could help, and the approach works on existing websites and apps that were built for people, so each company doesn't need to build special tools.
For you, the practical stakes are speed, cost and trust. The filing describes a safety checker, a way to ask you to confirm certain actions, and logging in only if you've given consent. It also says plans can be personalized from your profile and past tasks, which is convenient and also means the assistant knows a lot about you.
This is the 26th Microsoft patent we've tracked since July in our AI agents acting for you watchlist, following applications like button labels for AI and an AI nurse coach.
Of everything in this filing, this is close to the easiest kind of idea to turn into a product, because it needs no new hardware. Every piece it describes already exists: a browser that runs without a screen, an automation tool, a general AI model, and a plain data file to hold the map.
The hard part is upkeep. Websites change their layouts all the time, and a map of last month's checkout page can go stale. The filing answers this by checking for a telltale phrase before each click, then falling back to a screenshot and fixing the map when the phrase is missing. That gives the idea a believable path to working on a handful of popular sites first.
Trust is the other gate. Handing an AI your bank login takes more than clever plumbing, and the filing leans on consent prompts and a safety checker. Early versions would likely stay narrow and ask you to confirm a lot.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Microsoft patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
9 drawing sheets from US 2026/0339489 A1 · click any drawing to enlarge
Show the other 1 sheets
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in