Nvidia Patents a Way to Merge Radar and Camera Feeds Into One 3D Map for Self-Driving Cars
Cameras see detail but struggle in fog and darkness. Radar punches through weather but can't tell you what it's looking at. Nvidia's new patent describes a system that feeds both into a single AI and lets it figure out the full picture.
How Nvidia's camera-radar fusion actually works in a car
Today's self-driving systems almost always use cameras and radar separately, then try to reconcile two different readings of the world, a clunky process that can miss things or cause false alarms. Nvidia's patent describes a way to combine those two data streams much earlier, before any decisions are made, so the AI works from one unified picture instead of two incomplete ones.
The system takes what each sensor sees, converts both into a shared overhead view (think of it like a top-down map of everything around the car), and then lets a neural network analyze that combined map. The result is a grid showing exactly which spots in the car's environment are occupied by something solid, along with labels for what that something probably is.
The goal is a vehicle that can act more confidently in bad weather, low light, or cluttered traffic, because it isn't relying on just one type of sensor to make a life-or-death call.
… generating, using one or more neural networks and based at least on fusing the one or more first BEV features with the one or more second BEV features, output data indicating whether one or more voxels associated with an environment are occupied; …
Translation: Neural networks combine camera and radar data to figure out if specific 3D points in the environment contain objects.
How the two sensor streams get combined into one 3D picture
The patent describes a multi-modal fusion network, meaning a single AI model that ingests data from two completely different sensor types at the same time.
- Camera encoders process image frames and convert them into a Bird's Eye View (BEV) representation, a flattened, overhead-perspective feature map that strips out the camera's normal perspective distortion.
- Radar encoders do the same thing with radar returns, which carry distance and velocity information that cameras can't provide.
- The two BEV feature maps are then fused (mathematically combined) so that the strengths of each sensor fill in the blind spots of the other.
- A final neural network reads the fused map and produces a 3D occupancy grid, essentially a three-dimensional checkerboard where each cell is marked occupied or empty, plus a classification label (car, pedestrian, cyclist, etc.).
The output feeds directly into the vehicle's planning and control stack (the software that decides when to brake, steer, or accelerate). Because the fusion happens at the feature level rather than after each sensor has already made its own guess, the system can resolve ambiguous situations, a pedestrian in heavy rain, for example, that would trip up a camera-only or radar-only approach.
The patent covers the method broadly, meaning the architecture could run on Nvidia's existing automotive AI hardware platforms.
… multi-modal fusion networks for occupancy detection in autonomous and/or semi-autonomous systems and applications is described herein.
Translation: Self-driving vehicles use combined sensor systems to detect objects around them.
What this means for the future of self-driving perception
For anyone who drives, or who shares roads with vehicles that drive themselves, the reliability of object detection is everything. A car that mistakes a stationary truck for empty road, or fails to spot a cyclist because of glare, is dangerous. Fusing radar and camera data at a deep level, rather than just comparing their outputs at the end, is one of the main ways researchers believe perception accuracy can keep improving without adding entirely new sensor types.
Nvidia's interest in autonomous-vehicle perception shows up repeatedly in its patent filings, and this one sits squarely in that thread. The practical impact, if the approach performs as described, would be vehicles that handle edge cases more reliably, which is ultimately what separates a demo from a deployable product.
Nvidia's 59th filing we've tracked since May in the self-driving sensing race builds on one about merging overlapping sensor data and one about keeping cars in their lanes.
The cameras and radar sensors this system relies on are already standard equipment in many modern vehicles, which means the path to a real product runs through software and training rather than new hardware. That is a meaningful shortcut.
The remaining work is proving the system handles bad days well, the rare combinations of weather, lighting, and road conditions that expose the limits of any sensor blend. The document describes the architecture but does not address how that validation gets done.
If the safety testing holds up, this moves from research into something a manufacturer could plausibly deploy on equipment already rolling off assembly lines.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
27 drawing sheets from US 2026/0279030 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →