Nvidia Patents a Self-Driving Brain That Runs Two Decision Paths at Once
Most self-driving systems pick a lane: follow strict rules or trust an AI. Nvidia's new patent says you don't have to choose, and that running both approaches in parallel might be the safer bet.
How Nvidia's two-path driving system makes decisions
You're in the back seat of a self-driving car. The system has to decide whether to change lanes. Should it follow a fixed rulebook, or trust the AI that has studied millions of real-world drives? Most systems pick one approach. Nvidia's patent describes a system that runs both at the same time.
The idea is to have two "advisors" watching the same live view of the road. One advisor follows classical, rule-based logic, the kind you could write down as "if X, then Y." The other is a neural network, meaning it learned its instincts from enormous amounts of training data rather than hand-written rules. Both advisors produce a suggested action, and the system picks from them.
The clever part is that both advisors share the same underlying "perception" layer, the part that reads the sensors and builds a picture of the environment. That shared foundation is trained first, then locked in place while the AI advisor is fine-tuned on top. Nvidia keeps filing on autonomous-machine learning infrastructure, and this patent fits squarely into that work.
… generate, based at least on processing the scene embedding using a modular signal path comprising a rule- or heuristic-based planner, a first candidate action; generate, based at least on processing the scene embedding using an end-to-end signal path comprising a neural planner, a second candidate action …
Translation: The system calculates one driving choice using traditional programming rules and a second choice using a neural network.
How the hybrid stack trains and switches between paths
The patent describes a processor-level architecture with two distinct signal paths feeding off a single shared backbone network (also called an encoder or feature extractor). Think of the backbone as the eyes and ears of the system: it reads raw sensor data and converts it into a compressed, structured "scene embedding," essentially a rich internal snapshot of what the vehicle is currently experiencing.
From that shared snapshot, two paths branch off:
- Modular signal path: a rule-based or heuristic planner that applies engineered logic to produce a first candidate action. This is deterministic and auditable, you can trace exactly why it chose to brake.
- End-to-end (E2E) signal path: a neural planner trained on real-world driving data to produce a second candidate action. This path is less explainable but potentially more adaptive to novel situations.
Training happens in two stages. First, the backbone is pre-trained across all the data sources at once, including object detection tasks and the neural planner, without letting the classical algorithms interfere. Second, the neural planner is fine-tuned while the backbone weights are frozen (held constant), preserving what was learned in stage one. This two-step approach lets teams migrate a legacy rules-based system into a hybrid one without starting from scratch.
The final control decision can come from either path, or from a combination, depending on conditions. The architecture is designed for autonomous vehicles, semi-autonomous vehicles, and robots.
In a second training step, the neural planner may be fine-tuned with the common feature extractor or backbone network frozen.
Translation: Engineers train the neural network part while keeping the underlying visual processing system locked in place.
What this means for self-driving reliability and safety
For anyone who has followed self-driving development, the core tension here is real. Rule-based systems are predictable and easy to audit after a crash, but they struggle in situations their designers didn't anticipate. Neural systems handle novelty better but can fail in ways that are hard to explain or catch in advance. This patent describes a way to keep both options alive inside a single system, which means regulators can still inspect the rule-based output while the AI path handles the edge cases.
The staged training approach also has a practical upside for companies with existing self-driving stacks: you don't have to scrap years of engineering work to add an AI planner. You can graft the new path onto the old backbone. Whether that makes the combined system more trustworthy or just more complex is the open question, and the patent doesn't settle it.
Nvidia's 68th application we've tracked since May in the self-driving sensing race builds on earlier work like one using two decision systems and one for AI training simulations.
Running both a rule-based advisor and an AI advisor in parallel means the system always has two answers, and someone has to pick one. This patent says little about how that choice gets made, which is actually the harder problem than building either advisor in the first place.
The two-stage training approach is the choice most likely to hold up over time. By locking down the shared perception layer before teaching the AI planner on top of it, Nvidia makes it possible to update the AI side later without accidentally breaking the rules-based side. In a regulated industry, that separation matters because it narrows what has to be re-approved every time the system improves.
The cost is that the switching logic, the part that decides which advisor wins in a given moment, is left as an exercise for later. That deferred decision is not a footnote; it is where the real liability lives.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
14 drawing sheets from US 2026/0274297 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in