Get the newsletter
Google · AI / ML · Filed Jul 24, 2025 · USPTO published Jun 11, 2026 · Jun 11 release

Google Patents an AI That Reads App Screens the Way a Designer Would

Google is teaching an AI to look at an app screen the same way a designer or QA tester would — not just the pixels, but the underlying structure of every button, menu, and text field at once.

By · Updated · 3 min read

Google Patent: AI That Reads and Understands App Screens, figure from US 2026/0162416 A1
Figure from the official USPTO publication.
Publication number
US 2026/0162416 A1(US20260162416A1)
Applicant
Google LLC
Filed · Published
Jul 24, 2025 · Jun 11, 2026
Inventors
Yang Li, Xin Zhou, Gang Li, Mostafa Dehghani, Alexey Gritsenko
Status when we published
Waiting for an examiner
Document
20 claims
US classification
715/728
Parent application
Continuation of 17/812,208 (filed Jul 13, 2022)(17812208)

What Google's screen-reading AI actually does

Imagine hiring someone to audit every screen in your app for usability problems. They'd need to see what the screen looks like and also understand where every button and label actually sits in the layout. Google's patent is essentially training an AI to do exactly that, simultaneously.

The system takes a screenshot of any app screen along with a behind-the-scenes map of how the interface elements are arranged — think of it like a blueprint that lives underneath the pixels. The AI fuses both sources of information together before making any judgments about the screen.

The result is a single model that can handle multiple jobs at once: answering questions about the interface, labeling what elements do, spotting accessibility issues, or predicting what a user might tap next. Instead of training a separate AI for each task, Google's approach trains one shared brain that tackles all of them.

How the two-channel transformer fuses pixels and layout

The patent describes a multimodal neural network — a model that processes two different types of input at the same time — built around a component called an image-structure transformer.

Here's what goes in:

  • A screenshot processed by an image embedder (a module that turns pixels into a compact numerical description the AI can reason about)
  • A view hierarchy — the structured XML or JSON tree that Android and other platforms use internally to describe every UI element's type, position, and nesting — processed by a separate structure embedder

The image-structure transformer fuses these two streams using cross-tower attention (a mechanism that lets each stream look at and borrow information from the other, rather than staying siloed). This produces a single hidden representation — a rich internal description of the screen that captures both what it looks like and how it's built.

A second component, a question-answer transformer, then takes that representation and routes it through multiple task heads — specialized output layers, each trained for a different job like element classification, widget captioning, or tappability prediction. The whole network is trained end-to-end on a large dataset of labeled screenshots.

What this means for automated app testing and accessibility

For Google, this kind of model is a foundational tool for automated UI testing, accessibility auditing, and AI-assisted app development — all areas the company has been investing in through projects like the Android accessibility suite and AI-powered app crawlers. A single model that understands any app screen without needing app-specific training is a significant efficiency gain.

For you as an Android user, the downstream effect could be better accessibility features, faster detection of broken or confusing app layouts before they ship, and more capable AI assistants that can actually navigate apps on your behalf — something Google is already pursuing with features like Gemini's on-screen understanding.

Source: US 2026/0162416 A1 at the USPTO (PDF). Full patent text and figures are in the official publication.

Get our take in your Top Stories

Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Google patent news breaks.

Add Patentlyze on Google One click.
There are more where this came from

We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.

More from Google

All 1,079 Google patents →

More in AI / ML

All 2,637 AI / ML patents →