Nvidia Files Patent for Self-Driving Systems That Find Objects by Name
Most self-driving software only knows a short list of things it was taught to spot. Nvidia's patent filing describes a system that finds objects from laser scans using any word you give it, even ones it never trained on.
What Nvidia's find-it-by-name patent does
You're riding in a self-driving car when a tow truck appears half in your lane, partly hidden behind a van. Most car software only recognizes a short, fixed list of things it was trained on, so anything off that list is a problem.
Nvidia's filing describes a system built to skip that list. It takes the car's laser-scanner data, outlines every object in 3D (guessing at the hidden parts), and attaches a kind of meaning tag to each one. The car can then search those tags using plain words like tow truck or debris.
Need a new word? Add it. The filing says the vocabulary can be extended without retraining, and the labeling works from the laser scanner alone, with no camera feed at that moment.
… generate one or more labeled object instances based at least on querying the one or more semantic embeddings of the one or more detected object instances using one or more textual descriptions of one or more target classes …
Translation: It labels objects in the environment by matching their digital signatures with text descriptions.
How the network matches typed words to 3D shapes
The claims cover a three-step pipeline that starts with a frame of LiDAR data (a cloud of points from a laser scanner that measures the distance to surfaces around a machine):
- Detect. A neural network outputs predicted 3D masks (outlines of each object in a grid of small cubes called voxels) plus a semantic embedding for each one. That is a string of numbers capturing what the object is, in a space shared with language.
- Query. The system compares those embeddings against text descriptions of target classes, such as pedestrian or pallet, and labels the matches.
- Act. The machine's control software uses the labeled objects for navigation, grasping or manipulation.
Inside the network, a sparse generative encoder-decoder (a design that only computes on cubes it predicts are occupied, which saves memory) estimates the whole scene, including parts the scanner never saw. A transformer-based instance decoder then splits that scene into separate objects and produces one embedding per object.
Training needs no hand-labeled classes. The description mines pseudo-labels from driving footage: video segmentation models track objects across camera frames, the outlines are lifted into 3D, stacked over time, and cleaned up using LiDAR. At run time, no camera is needed. Dependent claims cover classes missing from training and word lists chosen by driving scenario.
… enabling zero-shot prompting at test and/or inference time. As such, the neural network may produce semantic features that may be used for open-vocabulary querying directly from 3D measurements …
Translation: The system can instantly recognize brand new objects using text prompts without needing prior training.
Why a swappable word list matters for robots
The practical payoff is flexibility. The filing says existing scene-completion techniques handle only about 20 classes of object, which leaves a car or robot guessing when it meets something unusual. With a word list that can be swapped, a car could look for construction barrier or debris on the road, and a warehouse robot could switch to pallet, forklift and box, without a fresh round of training.
The filing also describes loading different word lists for different situations, such as highway versus city driving, so the machine looks for what matters where it is. Claim 16 ties the idea to a machine with onboard chips, and dependent claim 17 names a chip rated at 275 trillion operations per second, which suggests the work is meant to run on the machine itself.
Nvidia's 87th filing we've tracked since May in the self-driving sensing race adds to a run that includes towing detection and move previewing.
Claim 1 is wide. It asks for three things: a neural network that turns laser-scan data from a machine into 3D object outlines plus embeddings, a step that labels those objects by matching the embeddings against typed descriptions, and the machine acting on the result. It names no network design, and the term "ego-machine" covers cars, robots and anything else that moves and senses.
The hidden-object filling that gives the patent its title lives in the description. Claim 1 asks only for predicted 3D masks. The firm edges are the LiDAR input, which claim 9 loosens to any 3D sensor, and the mandatory text-query step.
If granted, a claim this broad would reach any machine that labels objects from 3D scans by matching typed words and then acts on the labels. That covers a large share of where robot and car perception is heading. The idea is strong and the claim language is plain, which makes this one worth reading closely.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Nvidia patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
17 drawing sheets from US 2026/0321031 A1 · click any drawing to enlarge
Show the other 9 sheets
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in