Sony Patent Fills In Missing Game Frames Using 3D Scene Data
When your GPU can't keep up, Sony wants an AI to fill in the blanks, but unlike existing frame-generation tricks, this one actually knows what's hiding behind the scenery.
How Sony's frame-generation system handles hidden objects
Imagine you're playing a fast-paced game and a character ducks behind a wall, then steps back out. Your console has to redraw that character from scratch the moment they reappear, because it never had a chance to process what they looked like while they were hidden. That split-second redraw costs processing power, and if the console is already struggling, it can cause a blurry or choppy frame.
Sony's patent describes an AI system that watches each rendered frame and keeps a running memory of the entire 3D scene, including parts not currently on screen. When a new object or surface suddenly appears (because the camera moved or a character stepped out), the system already has a head start on what it should look like, pulling from the game's underlying 3D data rather than guessing from the previous image alone.
The result is that the AI-generated frames it inserts between real rendered frames are more accurate, especially at those tricky moments when something new pops into view. That means smoother motion without the visual artifacts that current frame-generation technology sometimes produces.
How depth maps reveal newly visible pixels each frame
The system works as a pipeline that processes frames in sequence, labeling each one by its position in that sequence (frame 1, frame 2, frame n, and so on). For every new frame, it runs two parallel tasks.
First, a machine learning model takes the current low-resolution or partially rendered input frame, combines it with a compressed memory of everything the AI has seen in all previous frames (called "accumulated feature information"), and produces both an estimated full-quality frame and an updated memory snapshot for the next cycle.
Second, the system compares depth maps (pixel-by-pixel distance measurements, like a 3D height map of the scene) from the current frame and the previous frame. Any pixel that was occluded (blocked by another object) in the last frame but is now visible is flagged as an "appearing pixel." These are the dangerous spots where a naive AI would guess wrong.
For those appearing pixels, the system pulls in virtual space information, raw data from the game engine about the 3D objects in the scene, their positions, and their surface properties. A second machine learning model combines the appearing-pixel map, the 3D scene data, and the accumulated memory to produce corrected "auxiliary information" that gets passed forward to the next frame's cycle. The loop repeats for every frame, keeping the AI's memory and its understanding of newly revealed geometry always up to date.
What this means for high-frame-rate PlayStation gaming
Frame generation, inserting AI-created frames between real rendered ones to hit 60 or 120 frames per second, is already shipping in PC graphics cards from Nvidia and AMD. The known weakness of those systems is exactly what Sony is targeting here: objects that appear suddenly from behind cover or at the edge of the screen tend to look smeared or ghosted, because the AI had no information about them a moment ago.
Sony Interactive Entertainment is the division behind PlayStation hardware and software, so a patent here points toward future console or cloud-gaming technology. If this approach ships in a PlayStation, it could mean that games run at higher frame rates on the same hardware, or that Sony's own game-streaming service delivers cleaner motion to players with slower connections, where the server-side renderer is always under pressure.
This is a technically specific patent aimed at a real, well-documented problem with current frame-generation systems. The idea of feeding 3D scene geometry directly into the AI correction step is a logical fix, and the depth-map comparison method for detecting newly revealed pixels is concrete enough to be meaningful. Whether Sony can make it fast enough to run in real time at console scale is the open question, but the patent is worth watching as a signal of where PlayStation's rendering roadmap is heading.
The drawings
7 drawing sheets from US 2026/0216598 A1 · click any drawing to enlarge
Which company should we read for you?
We track 17 companies here. Pro is the same weekly breakdown for any company you choose, delivered privately. Type a name and we'll scope it and send you a quote.
Get one Big Tech patent every Sunday
Plain English, intelligent commentary, no hype. Free.
Editorial commentary on a publicly published patent application. Not legal advice.