Sony Patents a Way to Freeze Any Game Scene and Explore It From Any Angle
Sony has filed a patent for a system that lets you pause a moment in a game and then replay it from a completely different camera angle, without the original game having to store that footage at all.
What Sony's scene-saving replay system actually does
You're playing a game and something incredible happens: a last-second comeback, a perfect shot, a wild boss fight finish. Right now, saving a clip means saving exactly what was on your screen, stuck at that one angle forever. Sony's patent describes a different approach: save the scene, and then rebuild it in three dimensions so you can watch it again from any direction you choose.
The system works during normal gameplay. When you decide to save a moment, the software generates a bunch of fake camera positions around the scene and renders what the scene would look like from each of those angles. It feeds all those images into an AI model, which uses them to reconstruct the scene's actual 3D geometry, almost like sculpting a physical model from photographs.
Once that 3D version is saved, you can pull it up later and fly your virtual camera anywhere around it, rewatch a moment from a bird's-eye view, or zoom in on a detail you missed. The game itself doesn't need any special camera system built in.
… generate a pseudo-viewpoint different from the viewpoint generate an image for learning corresponding to the pseudo-viewpoint; and generate three-dimensional scene information by machine learning using the image for learning as training data …
Translation: It creates fake camera angles and uses machine learning to figure out the 3D layout of the frozen scene.
How the AI reconstructs 3D geometry from fake camera angles
The patent describes a two-phase pipeline living on a content server that handles the heavy work on behalf of a player's device.
Phase one (capture): When a user triggers a save during gameplay, the system identifies the current frame and generates a set of pseudo-viewpoints, meaning camera positions that don't actually exist in the game's rendering engine but are mathematically derived from the scene's current state. The system renders images from each of those synthetic viewpoints and uses them as training data for a machine-learning model.
Phase two (reconstruction): The ML model processes those multi-angle images to produce what the patent calls 3D scene information, a structured representation of the scene's geometry, colors, and spatial relationships. The technique is similar in spirit to Neural Radiance Fields (NeRF), a method that infers 3D structure from 2D photographs taken from multiple angles. Once the 3D model is built, the server stores it.
Phase three (playback): When the user opens their saved scene later, the server uses the stored 3D information to render a new camera view from any arbitrary position, in real time. The first independent claim specifically covers the logic that generates pseudo-viewpoints, creates learning images from them, and then runs machine learning on those images to produce the 3D output.
In a saved scene viewing phase, the content server generates and outputs a display image representing the scene from an arbitrary viewpoint using the 3D scene information.
Translation: Once saved, you can look around the frozen game scene from any angle you want.
What this means for game clips and esports highlights
For ordinary players, this could make highlight clips far more expressive. Instead of a fixed screen-recording, a memorable moment becomes a scene you can scrub around, screenshot from the perfect angle, or share as a short cinematic. That's a meaningful upgrade over what current console share features offer.
For esports and game streaming, the implications are bigger. Sony's steady investment in capture and replay technology Broadcasters could use the same saved-scene data to produce replay angles that were never filmed, the kind of production tool that currently requires game developers to build dedicated spectator modes. The catch: the system relies on a content server doing the ML reconstruction, which means it needs network access, server capacity, and a game that cooperates with the pipeline. Offline play or fast-moving scenes with many dynamic objects could stress the reconstruction quality badly.
That makes this Sony's ninth filing in the AI vision patents we cover since May, a group that includes a game object styling patent and one reading ingredient flavors.
The design routes everything through Sony's servers, which means your saved scenes exist only as long as Sony decides to keep the lights on. For a beloved game still played years after its moment in the sun, that's a quiet but real fragility built into the foundation.
Reconstruction quality is the other honest cost. Converting a video clip into a walkable 3D space works beautifully on still, detailed moments, but fast action and chaotic scenes tend to blur and distort when you move the camera to an angle the original footage never captured.
Both risks are real, yet the core offer still reads as worth attempting: players get to freely explore any saved moment in three dimensions, without game developers having to build that capability into each title from scratch. The design just needs to know when to step back.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
20 drawing sheets from US 2026/0268598 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →