Sony Patents Tech That Rebuilds 3D Scenes From Ordinary Video Footage
Sony has filed a patent for a system that turns ordinary video into a fully navigable 3D scene, using a neural network that learns how light behaves at every point in space. The key trick is figuring out exactly where and which way the camera was pointing for every single frame.
What Sony's video-to-3D reconstruction system actually does
You're filming a sculpture or a room with a handheld camera, and the shot is slightly shaky. Sony's patented system watches your video and, frame by frame, figures out the precise position and angle of your camera at each moment.
With those positions known, a neural network is trained on the footage to build a full 3D model of whatever you filmed. The network learns what color light comes from every direction at every point in the scene, so that later, you can "walk through" it from any viewpoint, not just the ones your camera visited.
The system is designed to stop a common failure mode: when camera-position guesses are wrong, the 3D reconstruction falls apart. By using a short sequence of neighboring frames around each target frame to make each camera-pose estimate, the system stays consistent even when the camera moves quickly or unpredictably.
… a neural network that approximates a radiance field expressing a scene in a space including the object by a color in each of directions at each of positions in the space and opacity at each position in the space …
Translation: The system uses AI to map out the color and transparency of every point in a 3D space to recreate a scene.
How the neural radiance field learns from camera poses
The patent describes a two-part pipeline: camera pose estimation and neural radiance field learning.
Camera pose estimation is the process of working out where the camera was physically located and which way it was pointing for each video frame. Rather than analyzing each frame in isolation, the system looks at a short window of consecutive frames surrounding the frame in question. This gives the estimator temporal context (a sense of motion continuity) that makes its guesses more accurate.
Those poses feed into a neural radiance field (NeRF), a neural network that models a 3D scene as a function mapping any position-plus-direction to a color and an opacity value. In plain terms, the network learns: "if you stand here and look that way, you'll see this color." Once trained, it can render the scene from any viewpoint, including angles the camera never visited.
An alternative embodiment adds an encoding step: when filming, the camera generates additional metadata useful for pose estimation or NeRF training, then bakes that metadata into the video stream itself. This means the reconstruction pipeline can work with higher-quality hints even after the original footage has been compressed and stored.
A camera pose is estimated for each of frames of a moving image obtained by imaging of an object, and learning is performed of a neural network that approximates a radiance field expressing a scene in a space including the object …
Translation: The software analyzes video frames to figure out where the camera was positioned and builds a 3D model of the scene.
What this means for 3D capture in consumer devices
For consumers, the payoff is turning regular handheld video into something you can explore in 3D, without needing a specialized rig or multiple synchronized cameras. That feeds directly into augmented reality, volumetric video, and game asset creation pipelines that Sony has commercial interest in across its PlayStation and camera hardware businesses.
The frame-window approach to pose estimation is where this patent's practical edge lives. Shaky or fast-moving footage is exactly where NeRF pipelines usually degrade, so a more reliable pose estimator raises the ceiling for real-world use. For a broader read on where companies like Sony are placing their bets in spatial computing and AI imaging, Big Tech patent news covers the wave of neural rendering filings hitting the USPTO right now.
That makes this Sony's tenth filing we've tracked since July in our AI photo editing race, following one on copying a director's camera style and one on styling game objects from photos.
Claim 1 is narrower than the abstract makes it sound. It specifically requires that the camera-pose estimator use "images of some consecutive frames" around a target frame, not any single-frame or batch approach. That consecutive-window constraint is the claim's actual scope, and it's a meaningful one: any competing system that estimates poses frame-by-frame in isolation, or that uses a random sampling strategy, would fall outside this claim. In practice, that's a relatively precise foothold.
The NeRF space is crowded with academic and commercial work, and Sony's claim doesn't try to own NeRF training broadly; it owns a particular input strategy for pose estimation. That focus makes the claim easier to defend but also easier to design around. If granted, this could matter most in the device firmware layer, where camera hardware and video encoding happen before any cloud processing.
The alternative embodiment about encoding additional metadata into the video stream suggests Sony is thinking about this at the sensor and codec level, which is where its camera and PlayStation hardware teams actually operate.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
24 drawing sheets from US 2026/0253252 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →