Sony Patents Technology That Lets Viewers Replay Games From Any Camera Angle
What if every player's screen was teaching a server how to reconstruct a game world in three dimensions, so anyone could rewatch any moment from any angle? That is exactly what Sony's latest PlayStation-linked patent describes.
How Sony builds instant replays from players' own screens
Today, video game replays are locked to whatever camera angle the game engine recorded. If no one filmed a key moment from the right spot, that perspective is simply gone forever.
Sony's patent describes a server that collects the frames already being sent to all the players in a session and feeds them into an AI model in real time. Because hundreds or thousands of players are each watching the action from slightly different viewpoints, the server can stitch that crowd of perspectives together into a full 3D picture of the scene, the same way scientists reconstruct a 3D object from many photographs taken at different angles.
Once that 3D model exists, the server can render a replay from any viewpoint you choose, not just the ones anyone happened to record. It also tracks a heatmap showing where most players were looking, so broadcasters or spectators can see at a glance where the real action was.
… a 3D scene information generation unit for updating, at a predetermined rate, three-dimensional scene information indicating three-dimensional information associated with the 3D display world, by machine learning that uses the frames of the main image acquired from the plurality of client terminals aggregated as training data; …
Translation: The server uses gameplay footage from all players to constantly update a 3D digital model via machine learning.
How the server trains on live frames to rebuild 3D scenes
The patent describes a content server with four main components working in sequence during and after a game session.
During live play, the server does two things at once. It sends each player a rendered frame from their own viewpoint as normal, but it also collects those frames back as training data. The server runs a machine-learning process (specifically, it builds 3D scene information, a mathematical model of the game world's geometry and appearance) that updates continuously as new frames arrive from all connected players.
The heatmap layer is a parallel output: the server logs where each player's virtual camera is pointing at every moment and computes a density map showing which spots in the scene attracted the most attention. Think of it as a crowd-sourced heat map of the most-watched areas of the match.
During the replay phase, the server uses the 3D scene model to draw fresh frames from whatever viewpoint is requested. Because the underlying geometry is reconstructed, not just recorded, the viewer can orbit the scene freely rather than being tied to a stored camera track.
- Collects player-view frames as AI training images in real time
- Builds and continuously updates a 3D scene model from that crowd-sourced data
- Records a heatmap of player gaze or camera positions
- Renders free-viewpoint replay frames from the 3D model on demand
In a replay image distribution phase 322 , the content server generates replay images viewable from free viewpoints on the basis of the 3D scene information 324 , and outputs the replay images together with the heatmap (S 36 , S 38 ).
Translation: During the replay phase, the system builds new camera angles from that 3D data and sends them out with a popularity map.
What this means for esports broadcasts and game replays
For anyone watching esports or replaying a competitive match, the frustrating gap has always been you can only see what someone thought to record. A free-viewpoint system lets a broadcaster zoom into the exact angle that makes a clutch play readable, or lets a player review their own positioning from a coach's perspective, without requiring a dedicated server-side spectator camera that tracks every possible angle in advance.
Sony's steady investment in cloud gaming and remote rendering makes this fit neatly into a platform strategy where the heavy computation stays on Sony's servers and the client device just displays results. The heatmap feature is a quiet bonus: it gives tournament organizers concrete data on what audiences actually watch, which has real value for replay editing and broadcast direction.
That makes this Sony's 14th filing we've tracked in AI simulation since May, a topic that already covers work like one fixing corrupted motion data and one rebuilding 3D scenes from video.
The core idea here is genuinely practical. Games already send frames to players; the patent simply argues you should also read those frames back in, use them as a distributed camera rig, and let AI reconstruct the scene. That is a clever way to get free-viewpoint replay without embedding expensive volumetric capture hardware in every esports arena.
The heatmap detail is the part that might matter most to real users. Knowing where a crowd of players focused their attention is useful information for anyone editing a replay or designing a broadcast cut, and it comes for free as a byproduct of logging viewpoint data the server already handles.
The main open question is quality. AI-reconstructed 3D scenes from 2D frames can look convincing in controlled demos and patchy in fast, chaotic game environments. Whether this produces broadcast-grade results in a real match is an engineering problem the patent doesn't need to solve, but players and producers will care about it a lot.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
20 drawing sheets from US 2026/0268609 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →