Sony
Sony · AI / ML · Filed Dec 11, 2025 · USPTO published May 14, 2026 · May 14 release
Sony Patents an AI That Watches Your Game and Writes What It Sees
Sony is working on an AI layer that watches gameplay footage and automatically writes descriptions of what's happening on screen — think auto-subtitles, but for the visual action itself.
- Publication number
- US 2026/0134032 A1(US20260134032A1)
- Applicant
- Sony Interactive Entertainment Inc.
- Filed · Published
- Dec 11, 2025 · May 14, 2026
- Inventors
- Sudha Krishnamurthy, Justice Adams, Arindam Jati, Masanori Omote, Jian Zheng
- Status when we published
- Waiting for an examiner
- US classification
- 382/181
- Parent application
- Continuation of 18/138,620 (filed Apr 24, 2023)(18138620)
What Sony's AI scene annotation system actually does
Imagine watching a cutscene in a game and having your console quietly figure out: "a warrior is charging toward a castle gate at dusk." No human writer needed. That's the core idea here.
Sony's patent describes a system that takes individual video frames from a game or other audio-visual content and runs them through two AI components. The first figures out what's in the scene — characters, objects, environments. The second figures out what's happening — the action unfolding across one or more frames — and puts both into plain-language descriptions.
The system sits alongside other modules for things like acoustic effects and graphical style changes, suggesting it's designed as one piece of a larger content-enhancement pipeline. Whether that's for accessibility features, streaming overlays, or something else entirely, the patent doesn't say outright — but the building blocks are clearly there.
How the dual neural network pipeline captions frames
The patent describes a controller-coupled pipeline with two main AI modules working on incoming image frames from a host system (like a PlayStation console or cloud game server).
The Scene Annotation Module handles static understanding of a single frame. It uses:
- A first neural network that converts a raw image frame into a feature vector (a compact numerical representation of what the image contains)
- A second neural network that takes that feature vector and generates a natural-language caption describing the scene's elements — objects, characters, environments
The Action Description Module handles the temporal layer — it looks across one or more frames to recognize motion and activity, then generates a description of the action taking place. This is a meaningfully different task from scene captioning: understanding that a character is running or attacking requires tracking change over time, not just reading a single snapshot.
The patent also references companion modules — an Acoustic Effects Annotation Module, an Audio Tag Module, and a Graphical Style Modification Module — suggesting this annotation system is intended to plug into a broader content-enhancement or post-processing pipeline rather than operate as a standalone tool.
What this means for PlayStation accessibility and content tools
The most immediate application here is accessibility. Real-time scene and action descriptions are exactly what screen readers and assistive audio features need to make games playable for visually impaired users. Sony has been investing in accessibility tooling, and this kind of automatic visual narration could significantly lower the cost of building those features into games.
Beyond accessibility, auto-generated scene annotations have value for content moderation, streaming metadata, and AI training datasets. If your console can describe every frame of gameplay, that data becomes useful for search, highlights generation, and potentially feeding back into game AI systems. It's quiet infrastructure work, but the downstream applications are wide.
Source: US 2026/0134032 A1 at the USPTO (PDF). Full patent text and figures are in the official publication.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Sony patent news breaks.
Add Patentlyze on Google
One click.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
More from Sony
All 707 Sony patents →
More in AI / ML
All 2,637 AI / ML patents →
IBM
IBM Patent Teaches AI Models to Learn From Their Own Probability Uncertainty
Disney
Disney Patent Covers Animated Robots That Continuously Rethink Their Own Movement Goals
Adobe
From Flat to Layered: Adobe Patent Automates Design File Creation
Nvidia