Nvidia Files Patent for Turning Shaky Phone Videos Into Explorable 3D Scenes
Nvidia has filed for a method that turns a long, bumpy phone video into a 3D scene you can view from angles the camera never saw, with no camera data supplied up front.
What Nvidia's phone-video-to-3D patent does
Ever filmed a walk down a street on your phone and wished you could see the same scene from a spot where you never stood? That is the trick Nvidia's patent application is after: turning an ordinary, bumpy video into a 3D scene you can look at from new angles.
The hard part is that a phone video doesn't record exactly where the camera was at each moment. The system works that out on its own, frame by frame, while it builds the 3D scene. It also stores the scene in a compact way, spending detail on busy areas (the filing's example is a fire hydrant) and skipping empty space, so a long video doesn't swamp the computer's memory.
Once that's done, you pick a new viewpoint and the system draws what you would see from there. The filing says the clip can be anything from five minutes to an hour long.
obtaining an input video comprising a plurality of frames; training the 3DGS framework to obtain training outputs comprising estimated camera poses for each of the plurality of frames …
Translation: The system starts with a regular video and figures out where the camera was for every single frame.
How the octree anchors shrink a long video
The system takes a video with no camera information attached. It starts with just a handful of frames (the description gives the first three as an example) and uses a 3D reconstruction model to turn them into point clouds, which are swarms of dots marking where surfaces sit in space. From those it also estimates the camera's position and angle for each frame, called the camera pose.
The scene is then stored as octree anchors. Space is cut into cubes, and any cube crowded with dots is split into eight smaller cubes, repeating until the detail fits. Nearly empty cubes are dropped. Each surviving cube keeps a center point, a few offsets, a feature description and a scale value. A small neural network (a compact pattern-learning program) turns those into the color, opacity, size and rotation of fuzzy 3D blobs called Gaussians, generated only when needed.
Then frames are added one at a time:
- Match the new frame to the previous one to estimate the camera pose, then refine it by comparing rendered and real images.
- Add anchors for newly visible areas.
- Tune only the anchors visible nearby, then run a pass over everything. If the pose guess fails, re-optimize all earlier frames and retry.
To view a new angle, you supply a camera pose, and the system decodes the anchors in view and renders the image.
… an inference input indicating an inference camera pose may be obtained and an inference output from a viewpoint indicated by the inference camera pose may be generated.
Translation: You can then pick a brand new camera angle and the software will render what that view should look like.
What this means for 3D scenes from phone clips
If this kind of method reaches products, your ordinary phone clips could become 3D scenes you can look around in, without special scanning gear. You would film the way you always do. The filing lists uses such as virtual reality, augmented reality, virtual tourism, preserving cultural sites, video stabilization, and mapping for real estate or walking navigation.
For Nvidia, the angle is computing. The claims cover running the process in a data center or the cloud and streaming the result to your device, and using the generated images to test or certify neural networks in robots and self-driving vehicles. Keeping memory use down matters here, since the filing says long videos are exactly where earlier methods ran into memory limits.
Nvidia's 75th filing we've tracked in AI simulation since May adds to a run that includes one giving 3D models realistic surfaces and one that simplifies robot training.
Every shortcut here has a cost. By skipping camera data, the system gives up the safety net careful 3D scanning relies on, so it must guess where the camera was and keep correcting those guesses. The filing's repeated cleanup passes and its fallback of re-checking every earlier frame show how much work that takes.
Discarding nearly empty space saves memory, yet any area the video barely showed will come out thin on detail. Building the scene frame by frame also means an early mistake can ripple forward before a later pass catches it.
Still, the trade looks sensible. People filming a walk on a phone won't supply measurements, so a method that tolerates bumpy, long clips matches how the footage actually looks. The reported results come from the company's own tests, so they stay the company's account until others check them.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Nvidia patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
15 drawing sheets from US 2026/0320906 A1 · click any drawing to enlarge
Show the other 7 sheets
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in