Nvidia Patents a Multi-Camera System That Tracks Objects Across an Entire Space in 3D
Keeping track of a single person across dozens of cameras in a busy space is a problem that stumps most systems today. Nvidia's new patent describes a way to do it in 3D, in real time, using a neural network that pulls together footage from every angle at once.
How Nvidia's multi-camera tracking actually works
Every time a security guard reviews footage of a crowded airport, they face the same problem: one camera loses sight of a person, and finding them again on a different camera takes real effort. A computer that tries to automate this runs into the same wall.
Nvidia's patent describes a system designed to break through that wall. Multiple cameras watching the same space each send their footage into a neural network (a type of AI trained on examples). The network figures out that the person in camera 3 is the same person who just walked out of frame in camera 7, even if the angles are totally different. It does this by building a kind of 3D fingerprint for each person or object it sees.
The key trick is that the system also checks how clearly each camera can see a given object before deciding how much to trust that camera's view. A camera with a blocked sightline gets less say in the final result. The output is a continuous, real-time log of where every tracked object is and where it came from.
… determine a temporal alignment of the individual sensor image data based at least on an alignment threshold to define a batch of image data …
Translation: It syncs up the video feeds from different cameras so they match in time.
How the neural network fuses camera views into one 3D picture
The system takes in live image feeds from multiple cameras covering a shared space. Before doing anything else, it checks that the frames from different cameras are temporally aligned (meaning they represent roughly the same moment in time, within a set tolerance). Frames that are too far apart in time get held or discarded, so the AI is always working with a coherent snapshot.
Once a batch of synchronized frames is assembled, one or more neural networks process them together. The networks infer several types of data about detected objects:
- 3D bounding shapes: a box drawn around the object in three-dimensional space, not just a flat rectangle on a single camera's screen.
- Instance features: visual details that help distinguish one object from another (think clothing color, body shape, size).
- ReID feature embeddings: a compact numerical description of an object (a kind of fingerprint) that the system uses to recognize the same object when it reappears in a different camera's view. ReID stands for Re-Identification.
The ReID step is where the patent gets specific. Each camera produces its own view of an object. A dedicated ReID module combines those views, but it weights each camera's contribution by a visibility score (how clearly the camera can actually see the object at that moment). The resulting fingerprint is more reliable than one from any single camera alone.
The end product is a continuous track for each detected object across the whole monitored environment, even as objects move in and out of individual camera frames.
What this means for surveillance and robotics
For anyone building or buying large-scale camera systems (airports, warehouses, sports arenas, robotics fleets), reliable cross-camera tracking is one of the hardest unsolved pieces. Most systems today either work in 2D or require expensive hardware like lidar to get depth information. A pure-camera 3D approach could make capable tracking far cheaper to deploy.
Nvidia's interest in AI-powered physical-world perception has shown up across several domains, and this patent fits that pattern. The same core technology that tracks a person in a camera network could also track objects for a robot or an autonomous vehicle, which broadens its potential reach well beyond traditional security applications.
This is the 61st Nvidia filing we've tracked in AI simulation since May, adding to work like one that self-tests simulations and one guiding robot arms from text.
The cameras and computing hardware needed here already exist, which means turning this document into a real product is mostly a question of writing the software and training the models on enough real-world footage. No new chips, no new sensors required, just better instructions for equipment customers can already buy.
The cleverest part of the design is that it does not treat every camera as equally reliable. If one camera has a bad angle or a dirty lens, the system accounts for that automatically rather than letting one poor view corrupt the whole picture.
The remaining distance to a shipping product is field testing in difficult conditions: rush-hour crowds, dim parking lots, people constantly stepping in front of each other. The approach is sound, but only that kind of punishment reveals whether the system is ready for customers to depend on.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
11 drawing sheets from US 2026/0279058 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →