Nvidia Files Patent to Help Self-Driving Cars Keep Track of the Same Pedestrian or Car
To brake sensibly, a self-driving car has to recognize the pedestrian in this frame as the same one from the last frame. Nvidia's filing has the car check that against both the flat camera picture and the 3D scene at once.
What Nvidia's two-view object tracker does
Every time a car's cameras snap a new frame, its software has to answer the same small question: is that person on the curb the same one I saw a split second ago? Get it wrong and the car may think a pedestrian vanished, or that two different people are one.
Nvidia's patent filing describes a tracker that keeps two running guesses about every object. One is flat: where the object sits in the camera picture. The other is 3D: where it is in the real world and how fast it's moving. When the next frame arrives, the system predicts where each object should be, then checks new sightings against both guesses at once.
Why bother? A camera is decent at spotting things in a picture but shaky at judging distance. If the depth guess jumps around, the flat view can keep the match steady, and the reverse is true too.
… determine, based at least on the first 2D location, the first 3D location, the second 2D location, and the second 3D location, that the detected object corresponds to the tracked object; and perform one or more planning, navigation, or control operations based at least on the detected object corresponding to the tracked object.
Translation: The vehicle figures out if a newly seen person or car is the same one it was already following, then adjusts its driving.
How the tracker matches a new sighting
The filing describes a tracker that stores two sets of facts for each object it follows (a track). The 2D set covers where the object sits in each camera image: a box around it, a record of how that box shifted and grew, and a list of feature points (distinctive spots, like corners, that can be found again). The 3D set covers shape, position, velocity and acceleration in the real world.
When a new frame arrives, the system predicts both views. For 2D, it compares matching feature points between frames to work out how much the box slid and scaled. For 3D, it uses a Kalman filter (a standard math tool that blends a motion prediction with noisy measurements) to move the object forward in time.
Then comes matching. A neural network (software trained on many examples) finds objects in the new frame, and each detection gets a cost, where lower means a better match:
- 2D cost: how much the predicted box overlaps the detected box, plus feature-point comparisons.
- 3D cost: the gap in position and in viewing angle.
- Blend: a weight mixes the two, and it can change with distance, angle and whether the object is blocked.
A gating step throws out silly matches, such as a car paired with a person. Tracks are confirmed after enough sightings (three out of five frames, in one example) and dropped after enough misses. The results feed planning, navigation and control.
The systems and methods may then determine a 2D estimated location using the first 2D detected location and a 3D estimated location using the first 3D detected location.
Translation: The software predicts where the moving object should be next based on flat images and depth data.
What this means for self-driving braking
For anyone riding in or near a camera-based driver assistance system, the payoff is steadier behavior. The filing says earlier trackers leaned on 3D position alone, and camera-based depth guesses are less precise than radar or lidar. When a depth guess suddenly jumps, those trackers can lose the object. Using the flat image as a second check is meant to keep tracking steady and the speed estimate correct.
The claims cover a machine with CPUs, GPUs and hardware accelerators, plus a system-on-a-chip version. The text lists cars, robots, drones and warehouse vehicles as places it could run. That fits Nvidia's position as a chip maker whose hardware does this kind of work. As a patent application, it describes a method, not a product on sale.
This is the 103rd Nvidia filing we've tracked since May in the self-driving sensing race, joining earlier applications like one on filtering key vehicles and one on reading turn signals.
You would notice this feature mostly when it fails: a car that brakes for a ghost, or one that forgets a pedestrian stepping out from behind a parked van. Giving each object a steady identity from frame to frame is what prevents those moments.
The practical benefit is smoother driving. Fewer false alarms means fewer sudden stops that jolt you in your seat, and fewer missed objects means fewer near misses. In a driver assistance system, that shows up as emergency braking that fires when it should and stays quiet when it shouldn't.
Nobody will see this on a spec sheet, and the filing offers no test numbers, so how much better it performs remains unknown. It targets a failure you can picture clearly, a car losing track of a person, and that makes it a sensible thing for a chip company to protect.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Nvidia patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
23 drawing sheets from US 2026/0340865 A1 · click any drawing to enlarge
Show the other 15 sheets
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in