Nvidia Patents Software That Tracks Multiple Moving Objects Through 3D Space

By Patentlyze Team · Updated Jul 3, 2026

Keeping track of dozens of moving cars, cyclists, and pedestrians at once is one of the hardest problems in self-driving technology. Nvidia's latest patent describes a neural network approach that assigns each object its own persistent identity and follows it through a 3D scene frame by frame.

How Nvidia's object-tracking system watches a crowded scene

Imagine you're watching a busy intersection through a security camera. A red sedan enters from the left, a cyclist cuts across the middle, and a delivery truck backs out of a parking spot. A human brain keeps track of all three objects without losing them. Teaching a computer to do the same thing in real time, reliably, is genuinely hard.

Nvidia's patent describes a system where the AI processes a camera image and immediately identifies every distinct object it can see, giving each one its own compact data summary (called an embedding). It then uses special "track queries" to check whether each newly spotted object is the same one it saw a moment ago, building a running timeline for each object's movement through the scene.

The word "decoupled" in the patent title is key: the system separates the job of detecting objects from the job of remembering which object is which. That separation is designed to make the whole process faster and more accurate, especially in crowded, fast-moving environments.

How track queries link objects across video frames

The system works in a pipeline built around a transformer neural network (the same class of architecture behind large language models, but applied here to visual data instead of text). Here's the flow:

• Encode the image: The camera frame is processed into a compact numerical representation called "encoded image features" that captures shapes, positions, and depth cues in 3D space.
• Generate object embeddings: The network then produces a separate data vector (an "embedding") for each distinct object it detects in that frame. Think of each embedding as a fingerprint for one car, one pedestrian, or one cyclist.
• Match with track queries: Pre-assigned "track queries" represent objects the system has been following from previous frames. The network compares each new embedding against the existing track queries to decide whether a new detection is the same object it already knows about (an "association").
• Build trajectories: Once associations are confirmed, the system stitches them into a continuous path showing where each object has been and where it's heading.

The "decoupled" design means the detection step and the tracking step run somewhat independently, which the patent argues reduces errors that creep in when both tasks are tangled together in a single pass.

What this means for self-driving cars and robotics

For autonomous vehicles and robotics, losing track of a pedestrian for even a fraction of a second can cascade into a wrong prediction about where that person will be next. Current systems sometimes confuse two similar-looking objects or drop a track when something temporarily passes out of view. Nvidia's approach, by separating detection from tracking and using transformer attention to match objects across time, is aimed at making those failures rarer.

Nvidia already supplies the core computing hardware inside most self-driving development programs through its DRIVE platform. A more reliable 3D tracking algorithm built on the same transformer architecture the industry is already investing in would fit directly into that stack, and could also apply to warehouse robots, sports analytics, or any other setting where following many moving objects at once is the core challenge.

Editorial commentary on a publicly published patent application. Not legal advice.