Nvidia Patents a Way for Self-Driving Cars to Map Intersections in 3D From Flat Camera Footage
Intersections are where most serious crashes happen, and they're also where self-driving cars struggle most. Nvidia has filed a patent for a neural network that can read the full three-dimensional shape of an intersection using nothing but ordinary flat camera images.
How Nvidia's intersection-reading AI actually works
You're riding in a self-driving car as it approaches a busy four-way intersection. There are curbs, crosswalk corners, lane markings, and traffic signals all competing for the car's attention. The car's cameras capture all of this as flat, two-dimensional images, but to navigate safely, the car needs to know where everything sits in actual three-dimensional space.
Nvidia's patent describes an AI trained to take those flat camera frames and produce a full 3D map of what the intersection looks like. It does this by learning both what intersections look like in photos and what they must look like in the real world based on the rules of road geometry. So if the AI guesses that a curb curves in a way that's physically impossible, it gets penalized during training and corrects itself.
The result is a system that could give a self-driving car a much clearer picture of one of the most dangerous spots on any road, without needing expensive 3D sensors for every frame of input.
… generating, based at least on the one or more neural networks processing the image data, output data indicating one or more three-dimensional (3D) locations corresponding to the one or more features of the intersection …
Translation: The system calculates 3D spatial coordinates for intersection features using the neural network outputs.
How the neural network converts flat images to 3D road maps
The patent describes a deep neural network (DNN) that takes standard two-dimensional camera images of an intersection and outputs the three-dimensional coordinates of key features like lane boundaries, curbs, crosswalk edges, and stop lines.
Training this kind of network is tricky because the "ground truth" data (the verified correct answers the AI learns from) is typically available in 2D, not 3D. Nvidia's approach uses two separate loss functions (think of these as two different grading rubrics the AI is scored against during training):
- The first converts the AI's 3D predictions back into 2D and compares them against known 2D labels, so the AI can still learn from flat annotated data.
- The second checks whether the AI's 3D predictions obey the geometric rules of real-world roads. Intersections have predictable shapes: corners meet at angles, lanes have consistent widths, curbs follow smooth arcs. Predictions that violate these rules get penalized.
At deployment time, the trained network runs on live camera feeds from the vehicle and outputs 3D intersection geometry in real time. That geometry is then passed to the car's planning and navigation stack (the software that decides where the car goes and how fast) to inform safe, precise maneuvering through the intersection.
… a second loss function that analyzes the 3D predictions of the DNN in view of one or more geometric constraints—e.g., geometric knowledge of intersections may be used to penalize predictions of the DNN that do not align with known intersection and/or road structure geometries …
Translation: The AI is trained by punishing it when its predicted 3D maps violate the actual rules of real road layouts.
What this means for self-driving car safety at crossroads
Intersections are disproportionately dangerous, both for human drivers and for autonomous vehicles. A car that can accurately place road features in 3D space, rather than guessing from a flat picture, can make tighter, more reliable decisions about where to stop, when to yield, and how to track other road users.
What makes this approach notable is its reliance on camera images rather than dedicated lidar or depth sensors. Camera-only systems are cheaper and already widely deployed. If this technique works in production, it could make high-quality intersection awareness available on vehicles that don't carry the full sensor suite that current top-tier autonomous systems typically require. the pattern in Nvidia's autonomous-driving filings suggests the company is building out a complete software stack for vehicle AI, not just selling chips to run someone else's.
Nvidia's 76th filing we've tracked since May in the self-driving sensing race adds to a run that includes slow-camera detection and 3D parking view switching.
The system described here runs on ordinary cameras already built into most new vehicles, which means no new hardware has to be invented or installed before this can ship. That shortens the path to a real product considerably.
What still has to happen is the proving work. Every intersection type a driver might encounter would need to be tested until the system is reliable enough for safety regulators and insurers, and that process alone takes years of real-world miles regardless of how strong the underlying technology is.
Nvidia supplies the building blocks here; automakers and robotaxi companies would do that validation themselves before anything reaches a driver. This matters because it makes their eventual products meaningfully more capable, even if no finished feature is visible on any announced timeline.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
16 drawing sheets from US 2026/0285315 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in