A New Nvidia Patent Teaches Self-Driving Systems to Fill In What They Can't See
A self-driving car passing a parked van only sees the side facing the road, yet it still has to know where the rest of the van sits. Nvidia's patent application describes a way to train software to fill in that hidden part using video and laser scans, with no people drawing the labels.
What Nvidia's self-labeling vision system does
You're riding in a self-driving car, and it glides past a parked van. Its sensors only catch the side facing the road, so the car has to guess at the rest of the van's shape.
Nvidia's patent is about teaching software to make that guess well, without paying people to outline objects on thousands of scans. Instead, the system watches ordinary camera video, follows each object from frame to frame, and stacks what it sees over time into a fuller 3D shape. A laser-based depth sensor then fills in the gaps so the shape matches what is physically there.
Those automatically made labels train a model that later works from the depth sensor alone. You could then ask it, in plain text, to find things like a tow truck or debris, even if it never saw one labeled during training.
… refine the one or more 3D object masks based at least on binary occupancy data derived from 3D sensor data corresponding to the image data; and execute one or more iterations of training a neural network …
Translation: The system improves its 3D shapes using actual sensor readings to train its AI models.
How camera video gets turned into 3D labels
The pipeline starts with two streams of data from a calibrated vehicle: camera video and LiDAR (laser scans that measure distance and return a cloud of points). Segmentation models (AI tools that outline every object in a picture, such as Meta's Segment Anything) outline objects in a reference frame, and a video model follows each one across nearby frames. The result is an object track: a sequence of outlines for a single object.
From there, the steps go like this:
- Each flat 2D outline is lifted into 3D using the known alignment between the camera and the laser sensor.
- Outlines from many frames are merged into one 3D grid of tiny cubes (voxels), using the vehicle's known position, which builds denser object shapes.
- A binary occupancy map (a simple filled-or-empty record built from the laser scans) refines those shapes. A conditional random field (a method that nudges neighboring cubes toward the same label) spreads labels into occupied cubes the camera missed.
- Each sparse single scan is paired with its completed object shapes and a semantic embedding, a numeric fingerprint (such as CLIP's) that ties an object's appearance to words.
The filing says moving objects can be removed, motion-corrected, or left alone. The trained neural network then takes laser data only. At run time, text prompts are compared against each object's fingerprint to find matches.
… effectively combining shape completions and semantic information. As such, the resulting pseudo-labels may be used to train a neural network to perform scene-scale object-level completion.
Translation: It merges guessed shapes with object meaning to train AI that fills in missing parts of a scene.
Why skipping human labels matters for robots
If you ride in a self-driving car or work near a warehouse robot, the hidden parts of objects matter. A machine that only trusts what it sees this instant can misjudge where a parked truck ends or what sits behind a box. Training it to fill in whole shapes is aimed at safer navigation and grasping.
The text-query idea may matter even more. The filing says different driving modes, such as highway or urban, could load different word lists, while a warehouse robot could load pallet, forklift and box. New words like construction barrier could be added without retraining. And because the labels come from unlabeled recordings, the approach points toward learning from far more driving footage than people could outline by hand.
Nvidia's 90th filing we've tracked since May in the self-driving sensing race adds to a run that includes one on reading road bumps and one on budget radar mimicking lidar.
Claim 1 is wide. It lays out four steps: outline objects in camera images with segmentation models, merge those outlines into a 3D grid, tidy them with a filled-or-empty map from a 3D sensor, then train a neural network on the result. It never says car, never says laser scanner (it says "3D sensor data"), and never mentions the text-search feature that gets the most attention in the description.
That breadth means the claim, if granted, would reach any robot, drone or simulation that builds its training labels this way, whatever the machine. The semantic fingerprints and the label-spreading method appear only in later claims, so they add narrower layers on top of the core. Claim 17 is leaner still: it covers just the training step, as long as the labels were produced this way.
Every individual step uses existing tools, so the novelty sits in the combination. It is an ambitiously worded filing with a practical idea underneath: let cameras do the labeling, then teach a laser-only model to remember it.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Nvidia patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
17 drawing sheets from US 2026/0320803 A1 · click any drawing to enlarge
Show the other 9 sheets
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in