Nvidia Patents a Way to Merge Past and Present Camera Feeds in Self-Driving Cars
A self-driving car that only sees the world in single frozen snapshots will always be a step behind. Nvidia's latest patent filing is about teaching vehicles to layer what they're seeing right now on top of what they saw a moment ago, giving the car a richer mental picture of everything around it.
How Nvidia's self-driving cars piece together the road over time
Ever wondered how a self-driving car knows whether that blur at the edge of the frame is a cyclist speeding up or a plastic bag blowing across the road? A single camera frame often can't tell you, but a sequence of frames over time can.
Nvidia's patent describes a system where a vehicle's AI doesn't just look at what its cameras see right now, it stacks that image together with processed snapshots from recent moments, blending them into one combined picture. That combined picture is what the car then uses to decide where to steer, when to brake, and how to plan its route.
The key word here is fusion. Instead of treating each camera frame as a standalone observation, the AI is trained to weave multiple frames across time into a single, more informed view of the world around the vehicle. Think of it like the difference between judging a stranger's mood from one photograph versus watching a few seconds of video.
generate, using one or more machine learning models and based at least on fused feature maps associated with a plurality of time steps, a temporally fused feature map, the fused feature maps generated using sensor data obtained using the external sensors …
Translation: The system creates a single map by combining visual data from the car's sensors across several different moments in time.
How the model fuses sensor snapshots across multiple time steps
The system uses a machine learning model (MLM) to process raw image data from cameras placed around a vehicle. Each set of camera images is turned into a feature map (essentially a compressed numerical representation of what the cameras see, stripped of raw pixels and focused on meaningful patterns like edges, shapes, and distances).
Critically, the model doesn't just produce one feature map for the current moment. It also holds onto feature maps from previous time steps and runs them through dedicated layers designed to perform a temporal transformation (a mathematical adjustment that accounts for the fact that the car has moved since the earlier frame was captured, so objects need to be repositioned correctly before blending).
Once those historical feature maps are transformed and aligned, they are fused together with the current feature map to create a temporally fused feature map. That single combined map is what drives downstream decisions:
- Path planning (where should the car go next?)
- Navigation (is the route still clear?)
- Vehicle control (when to accelerate, brake, or turn?)
The whole process runs on a combination of CPUs, GPUs, and dedicated hardware accelerators inside the vehicle, suggesting this is designed for real-time, on-board inference rather than cloud-assisted processing.
Systems and methods are disclosed that use a machine learning model (MLM) to intrinsically fuse feature maps associated with different sensors and different instances in time.
Translation: The technology uses artificial intelligence to blend data from various cameras and past observations into one cohesive view.
What this means for autonomous vehicle perception pipelines
For passengers and pedestrians alike, the practical upside is a vehicle that makes fewer mistakes at the edges of its perception, particularly with objects that are partially hidden, moving fast, or only briefly visible. A system that remembers what it saw half a second ago can often figure out what's happening now with much more confidence than one working from a single frozen frame.
Nvidia already supplies the chips and software stack that power many of the leading autonomous and semi-autonomous driving programs, so a patent like this fits squarely into the company's push to own the full perception pipeline, from silicon to the AI model running on it. Autonomous vehicle perception is one of the more active areas across this week's Big Tech patents, and this filing shows how much engineering work still goes into something as apparently simple as "seeing the road clearly."
Claim 1 is written broadly enough to cover essentially any autonomous or semi-autonomous machine that fuses sensor-derived feature maps across multiple time steps and uses the result for planning, navigation, or control. That scope is wide. It doesn't tie the claim to a specific sensor type, a specific fusion architecture, or a specific transformation method, which means if granted as written, it could assert itself against a wide range of multi-camera temporal fusion approaches used in the industry today. The practical question is whether prior art in bird's-eye-view perception and recurrent feature fusion, which has been publicly published by academic and industry groups for several years, would narrow the claims during examination. What the claim clearly does cover is the full loop from sensor data to vehicle action through a temporally fused representation, and that end-to-end framing is the strongest part of the filing.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
14 drawing sheets from US 2026/0236563 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →