Nvidia Patents a Memory-Driven 3D Scene Reconstruction System for Self-Driving Cars

By Patentlyze Team · Updated May 29, 2026

What if your self-driving car could remember what a street looked like last Tuesday — and use that memory to fill in the parts of today's view that a parked truck is blocking? That's the core idea in Nvidia's latest patent filing.

How Nvidia's driving AI remembers what it can't see

Imagine you're driving down a street you've been down dozens of times before. A delivery truck is parked at the corner, blocking your view of the intersection. A human driver uses their memory of that intersection to make safer decisions — even without seeing it right now. Nvidia's patent is trying to give autonomous vehicles that same kind of spatial memory.

The system takes a camera image from right now and combines it with a saved 3D model of the same scene from earlier visits. Those older snapshots captured the street when it was clear, so the system can fill in the gaps that today's moving trucks or pedestrians are hiding.

The patent also describes using this combined 3D model to edit scenes — for example, removing a car from the image entirely, moving it to a new spot, or even transplanting objects from an older image into a newer one. That kind of scene editing is a powerful tool for generating synthetic training data for autonomous driving systems.

How past traversals fill in occluded scene geometry

The core invention is a pipeline that builds a 3D representation of a scene by fusing two kinds of input: a current image (or set of images) and a precomputed model of the scene's time-invariant components — meaning the parts that don't change, like road surfaces, buildings, and curbs — derived from earlier visits to the same location.

The prior-visit model acts as a kind of spatial memory. Because autonomous vehicles often traverse the same routes repeatedly, older sensor captures can reveal structures that are currently occluded (blocked from view) by moving objects like cars or pedestrians. By integrating this historical context, the system can reconstruct more complete 3D geometry even when today's view is partially blocked.

Once the combined 3D representation is built, the patent describes an editing step that lets the system manipulate objects within the reconstructed scene:

• Remove an object (e.g., erase a parked car)
• Relocate an object within the current image
• Transport an object from a historical image into the current one

This editing capability is especially valuable for synthetic data generation — the practice of artificially creating training scenarios to expose a model to rare or dangerous situations it might not encounter in real-world data collection. The underlying 3D scene model appears to be a variant of a neural radiance field (NeRF)-style approach, which represents scenes as continuous volumetric functions that can render novel viewpoints.

What scene memory means for autonomous driving datasets

For autonomous driving companies, one of the hardest problems is occlusion — the world is constantly hiding itself behind other objects. Most current reconstruction systems treat every new drive as a blank slate, which means anything blocked from the camera simply doesn't exist in the model. Nvidia's approach of integrating prior traversal data is a meaningful step toward more complete environmental understanding.

The scene-editing capability is arguably just as important as the reconstruction itself. Generating realistic, physically accurate synthetic training scenarios — especially ones involving edge cases like unusual vehicle placements or rare pedestrian behavior — is a major bottleneck for the industry. A system that can plausibly insert, remove, or relocate objects in a photorealistic 3D reconstruction could accelerate data augmentation pipelines significantly, reducing the cost and time of collecting and labeling real-world data.

Editorial commentary on a publicly published patent application. Not legal advice.