Sony Files Patent to Generate Real-Time Descriptions of Game Screen Content

By Patentlyze Team · Updated Jul 10, 2026

Sony has filed a patent for a system that watches a game's screen in real time and uses AI to continuously describe what's happening, frame by frame. That might sound simple, but the range of things it could power, from accessibility tools to live game commentary to in-game AI behavior, is genuinely broad.

What Sony's real-time game-screen AI actually does

Imagine if someone sat next to you while you played a video game and narrated everything happening on screen, not just once at the start, but constantly, as the game changed. Sony's patent describes a system that does exactly that, automatically.

A machine learning model watches the game's screen as it runs and produces a running description of what it sees. The model was trained by pairing game screenshots with text that describes those screenshots, teaching it the relationship between visuals and language. Once trained, it can generate those descriptions on its own, in real time, as the game plays out.

The patent doesn't lock that output to one specific use. Whatever the system generates can feed into any number of follow-up processes, things like generating audio commentary, informing an AI opponent, or making games more accessible to players who are blind or have low vision.

How the model links screen images to language descriptions

The core of the system is a machine learning model trained on pairs of game screen images and text descriptions of those images. During training, the model learns to map visual content to language. Once deployed, it watches live game output and produces descriptive text (or other language-based data) that reflects whatever is currently on screen.

The patent describes the process as sequential and continuous: the system doesn't analyze a single screenshot and stop. It keeps pace with the game, processing screen data as it changes and generating updated model output to match. This matters for games especially because the visual state can shift dramatically from second to second.

The phrase "predetermined processing" is deliberately vague in the patent, but it's the downstream step that receives whatever the model outputs. That could mean:

• Generating spoken audio descriptions for accessibility
• Feeding a game's AI opponents with information about the current state of play
• Powering a live commentary or highlight-generation system
• Triggering game events or achievements based on what happens on screen

The inventors are from Sony Interactive Entertainment, the division that runs the PlayStation platform, which points strongly toward gaming as the primary use case, though the claim language covers any content whose details change over time.

What this means for accessibility and game AI on PlayStation

Accessibility is the most immediate angle here. Screen readers exist for operating systems and apps, but gaming has lagged far behind. A system that continuously reads a game's visuals and converts them to language could power real-time audio descriptions for blind or low-vision players, something the PlayStation platform has not meaningfully offered at the system level. If Sony bakes this into the PlayStation hardware or software layer, developers wouldn't need to build it themselves.

Beyond accessibility, this is also a building block for more capable in-game AI. An AI opponent or companion that can read the screen like a player does, rather than relying on structured game data fed to it by the engine, would behave very differently from what exists today. That's a longer-term bet, but the patent infrastructure points in that direction.

Editorial commentary on a publicly published patent application. Not legal advice.