Nvidia Files Patent to Help Self-Driving Cars Identify Pedestrians From Distance Sensors
Nvidia's filing covers a self-driving AI that picks out pedestrians and cars in laser scans, and describes training it with labels people drew on ordinary camera photos.
What Nvidia's laser-scan detector actually does
Laser sensors on self-driving cars produce a cloud of dots, and it is surprisingly hard for a person to look at that cloud and say which dots are a pedestrian and which are a pole. Yet the AI that reads those scans has to learn from exactly those hand-drawn labels, so sloppy labels mean a less reliable car.
Nvidia's filing sidesteps the problem. People label ordinary camera photos, where a car or a pedestrian is easy to spot, and the system copies those labels onto the matching laser data. It then throws out stray dots that don't fit, such as a point that bounced off a building behind a person.
Once trained, the AI reads a flattened version of the laser scan, marks what each object is and where it sits, and maps that back into 3D. That picture goes to the car's driving software for planning and steering.
generate, using one or more neural networks and based at least on a converted representation of sensor data generated from an initial representation of the sensor data obtained using the one or more external sensors: one or more segmentation outputs corresponding to one or more objects; and one or more localization outputs corresponding to the one or more objects; …
Translation: Neural networks process sensor data to figure out what objects are nearby and exactly where they are located.
How photo labels get copied onto laser scans
The claims describe a vehicle or robot with standard computing hardware (CPUs, GPUs and dedicated AI chips) plus outward-facing sensors. It takes raw sensor data, such as a LiDAR point cloud (a 3D map of dots made by bouncing laser pulses off the surroundings), and converts it into another form, like a range image: a flat picture where each pixel stores distance, height and brightness.
A neural network (a pattern-learning program) reads that picture and produces two kinds of answers. Segmentation outputs say what each pixel belongs to, such as a car, pedestrian, cyclist or drivable road. Localization outputs say where a box around each object sits. The system matches the two sets of answers, projects the result back into the original 3D data, and uses the detections for planning, control or navigation.
The training side is where the filing gets unusual:
- Labels drawn on camera photos are carried into the laser data using the known positions of the camera and the laser sensor.
- Stray points are filtered using size limits, since all the points for one pedestrian should sit within a few meters of each other.
- Virtual laser sensors at different positions can create extra training scans from existing data.
Those training steps are described in the filing, but this application's claims cover the detection side, not the label copying. A separate claim covers a chip that plans a path from a top-down view built from two or more sensor types, including a LiDAR sensor rated ASIL B (an automotive safety grade).
For example, annotations generated in the image domain may be propagated to the LiDAR domain to increase the accuracy of the ground truth data in the LiDAR domain …
Translation: Labels drawn on camera images are carried over to the laser data, which makes its training labels more accurate.
Why cleaner training labels matter on the road
If you ride in a car with laser sensors, or share the road with one, what matters is how reliably it tells a pedestrian from a signpost. Better labels during training should mean fewer misses and fewer false alarms, like braking hard for something that was never there. Skipping hand-labeling of laser data also makes it cheaper to build the huge training sets these systems need.
The claims name a system-on-a-chip with CPUs, GPUs and AI accelerators, so the pitch pairs software with silicon. Nvidia already holds two patents on this idea: this application continues earlier ones granted as US 11,906,660 and US 12,730,190, with roots in a 2019 provisional filing. The description says the same approach could serve robots, drones, boats and warehouse vehicles.
That makes this Nvidia's 104th filing we've tracked since May in our self-driving sensing race, building on filings that keep track of the same pedestrian or car and focus on the few vehicles that matter.
You would notice this only when it works: a car that brakes for a real child and ignores a drifting plastic bag. Better training labels are how a system gets there, and the label-copying method the filing describes goes straight at the weak spot, since people can't draw accurate labels on a cloud of dots.
Nothing here changes what you do in a car. The payoff is indirect: fewer missed pedestrians and fewer needless stops, if the method holds up on real roads. The text we read gives no accuracy figures, so how much better it gets stays unproven.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Nvidia patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
17 drawing sheets from US 2026/0338917 A1 · click any drawing to enlarge
Show the other 9 sheets
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in