Get the newsletter
Microsoft · AI / ML · Filed Apr 25, 2025 · USPTO published Jun 18, 2026 · Jun 18 release

Microsoft's New Patent Teaches AI to Click Through Software the Way a Person Would

Microsoft is working on an AI model that learns how to click through software interfaces the same way a person would — by studying recorded sequences of steps taken to complete real tasks.

By · Updated · 3 min read

Microsoft Patent: AI Model Pre-Training for UI Navigation, figure from US 2026/0170817 A1
Figure from the official USPTO publication.
Publication number
US 2026/0170817 A1(US20260170817A1)
Applicant
Microsoft Technology Licensing, LLC
Filed · Published
Apr 25, 2025 · Jun 18, 2026
Inventors
Xiaoyi ZHANG, Yuwang WANG, Zhizheng ZHANG, Wenxuan XIE, Yan LU
Status when we published
Waiting for an examiner
Document
21 claims
US classification
715/708
Parent application
National Stage Entry of PCT/US2023/081484 (filed Nov 29, 2023)(PCTUS2023081484)
AI assistant & agent

What Microsoft's UI-navigation AI model actually does

Imagine you need to complete a task in a complicated piece of software — say, exporting a report buried three menus deep. A human does it by clicking through a series of screens in a specific order. Microsoft's patent describes a way to teach an AI to do that same kind of navigation, automatically.

The core idea is to train the AI not just on individual screens, but on full navigation paths — the whole sequence of steps from start to finish, matched with a description of what the task actually was. That way, the AI learns the relationship between a goal ("export the report") and the route through the interface to get there.

The goal is an AI model that can generalize: once trained, it should be able to handle new interfaces and tasks it hasn't explicitly seen before, without needing to be retrained from scratch each time.

How the model learns from recorded navigation paths

The patent describes a pre-training pipeline for an AI model focused on user interface navigation — the kind of task where an agent must move through screens, menus, and dialog boxes to complete a goal.

The training data has three parts working together:

  • Navigation paths: recorded sequences of UI screens that correspond to completing a specific task
  • UI descriptions: text descriptions of the elements visible on each screen (buttons, fields, menus)
  • Task descriptions: plain-language descriptions of what the navigation path is trying to accomplish

The feature extraction model (the AI component that converts raw UI and task information into a structured internal representation) is trained on the correspondence between all three. Rather than learning about individual screens in isolation, the model learns at the path level — meaning it understands how a sequence of screens connects to a real-world goal.

This approach is designed so the pre-trained model can be adapted to downstream navigation tasks (new software, new goals) without starting over, a technique common in modern AI training called transfer learning.

What this means for AI agents controlling your software

AI agents that can operate software interfaces are becoming a real product category — think of tools that automatically fill forms, book appointments, or dig through enterprise software so you don't have to. The limiting factor today is that these agents tend to be brittle: train them on one app and they fall apart on another.

By pre-training a model on navigation paths rather than static screenshots, Microsoft's approach could produce agents that transfer more reliably across different software products. That's directly relevant to Microsoft's push to embed AI agents inside products like Windows, Office, and Azure — where the AI needs to operate unfamiliar interfaces without constant retraining.

Source: US 2026/0170817 A1 at the USPTO (PDF). Full patent text and figures are in the official publication.

Get our take in your Top Stories

Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Microsoft patent news breaks.

Add Patentlyze on Google One click.
There are more where this came from

We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.

More from Microsoft

All 646 Microsoft patents →

More in AI assistant & agent

All 413 AI assistant & agent patents →