Sony Patents an AI That Rebuilds Your App's Interface While You Use It
Sony has filed a patent for a system that watches what you're doing inside an app and, in real time, asks an AI to replace the interface you're looking at with a different one that still does the same job.
What Sony's AI-swapped interface actually does for you
A person sits down with a PlayStation controller, opens an app, and the menus they see were designed for a big TV screen. Then they pick up their phone to keep playing, and the same menus look cramped and awkward. That's a real friction point for Sony, whose remote-play ecosystem spans televisions, phones, handhelds, and PCs.
Sony's patent describes a system where a remote server runs the app and watches what you do with it. When you interact, the server sends a snapshot of the current interface and your action to an AI. The AI decides whether a different layout would work better, and if so, sends a replacement interface back to your screen.
The key detail: the replacement must keep at least one core function of the original in place. You don't lose access to what you were doing. The AI isn't redesigning the app from scratch; it's swapping in a version that fits your situation better.
… generating an input to an artificial intelligence model based on the first GUI and a user input to the application at the device; receiving, from the artificial intelligence model and based on the input, an output indicating a second GUI …
Translation: The system feeds what you see and do into an AI to figure out how to redraw the screen.
How the AI picks and serves a replacement interface
The system has three moving parts working together on a remote server.
- App execution happens server-side. The application runs on Sony's infrastructure, not on your local device. Your phone or tablet is essentially a display and input terminal.
- The AI model gets two inputs: the current graphical user interface (GUI) the user is seeing, and whatever action the user just took. Together those two signals tell the model what the app is doing and what the user is trying to accomplish.
- The AI outputs a second GUI. This replacement interface must share at least one function with the original (meaning the patent requires functional continuity, not a full redesign). The server then pushes this new interface to the device.
The claim is deliberately abstract. It does not specify what kind of AI model is used, what the interface looks like, or what triggers a swap. The only concrete constraint is the functional overlap requirement: the new GUI can look completely different as long as it still lets you do at least one thing the original could do.
That functional-overlap rule is the heart of the patent. Without it, the claim would cover any AI that generates an interface, which would be impossibly broad. With it, Sony is staking ground specifically on AI-driven interface substitution during an active session.
What this means for PlayStation and remote play
For Sony, the most obvious application is PlayStation Remote Play, which already lets users stream PS5 games and apps to phones, tablets, and PCs. A consistent complaint is that interfaces built for a controller and a TV are clunky on a small touchscreen. A system that automatically adapts the layout on the server side, without patching the app itself, could smooth that experience without requiring developers to ship separate interface code for every device type.
Broader than gaming, the idea applies to any cloud-streamed application. If this approach works, the server becomes responsible for fitting the UI to the device, taking that burden off individual app developers. How well any AI model would actually do that job in practice is an open question the patent doesn't answer.
Sony's 23rd filing we've tracked in Enterprise AI since May follows applications like remote device repair and adding missing people to photos.
Claim 1 is strikingly broad. It covers any computer that runs an app remotely, feeds the current interface and a user action into any AI model, gets back a replacement interface that shares at least one function, and displays it. That one-function overlap requirement is the only real constraint, and it is thin: a replacement that keeps a single button does the job on paper.
If granted as written, this claim could create friction for any cloud-streaming platform that uses an AI to adapt or regenerate its interface during a session. That covers a lot of potential territory in cloud gaming, enterprise SaaS, and remote desktop tools. Whether the patent office would accept the claim without narrowing it is the real question, because prior art in adaptive UI and cloud rendering is deep.
Sony's track record in cloud-streaming patents suggests this is part of a broader push to own the infrastructure layer of remote play, not just the game content. The interesting fight, if this ever gets challenged, will be over whether that one-function rule is enough to distinguish it from a decade of existing adaptive-UI research.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Sony patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
7 drawing sheets from US 2026/0277630 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in