Nvidia Patents an AI That Builds Driving Simulation Scenes From Separate Lighting and Layout Data
Training a self-driving car takes millions of hours of road footage that don't exist yet. Nvidia's new patent describes a way to let neural networks generate those missing scenes on demand, splitting a driving image into two separate pools of information so each can be changed without breaking the other.
How Nvidia's driving simulator fills in missing frames
Imagine you're testing a self-driving car and you need footage of a rainy night on a highway overpass, but you only have sunny afternoon clips. Generating that scene from scratch is hard because the overall mood of a scene (lighting, weather, time of day) and the specific objects in it (cars, lane markers, barriers) are tangled together in every pixel.
Nvidia's patent describes a system that untangles those two things. One neural network pulls out the scene-wide information, things that apply everywhere in the image equally, like the color of the sky or how bright the sun is. A second set of processes handles the position-specific stuff, like exactly where a truck sits in the left lane. By keeping those two pools of information separate, the system can swap in a new sky or new weather without moving the truck.
The result is a driving simulator that can respond to a user's instructions and generate new video frames that look consistent with the original scene, but reflect whatever change was requested.
use one or more neural networks to generate a representation of spatially-independent information and a representation of spatially-dependent information based, at least in part, on user input and one or more first images of a driving simulation …
Translation: The AI splits driving scenes into separate layout and lighting data based on user input.
How the network splits scene-wide from object-level data
The patent covers a processor-level system that runs one or more neural networks to produce frames for a driving simulation. The key idea is a clean split between two types of information the network must track.
Spatially-independent information (things that are the same everywhere in the frame, like overall brightness, color temperature, or weather conditions) is extracted and stored separately from spatially-dependent information (things that vary by location, like the shape of a lane line or the position of a pedestrian). The network produces a compact representation of each.
When a user provides input, say, a steering command or a request to change the weather, the system feeds that input alongside both representations into the network. The network then generates a new image that shows what the scene would look like with that change applied, while keeping everything else consistent.
The claim covers the entire pipeline:
- Taking user input plus existing simulation frames as starting material
- Generating separate representations for scene-wide and location-specific content
- Combining those representations with the user input to synthesize new frames
This is sometimes called a world model in AI research, a system that learns the rules of an environment well enough to predict or generate what it would look like under new conditions.
… determine spatially-independent information and spatially-dependent information of the one or more images and process the spatially-independent information and the spatially-dependent information to generate the one or more spatially-independent features and one or more spatially-dependent features within the one or more images …
Translation: It analyzes the visual features to build new simulation scenes from the separated data.
What this means for AI-powered autonomous vehicle testing
For autonomous vehicle developers, the bottleneck has never been compute alone, it has been the sheer variety of driving conditions they need to test against. A simulator that can take real footage and reliably transform it into night driving, wet roads, or heavy traffic, without retraining from scratch, could cut the time between design iteration and safety validation significantly. Nvidia already sells hardware and software tools to this market, so this patent fits directly into that commercial stack.
Claim 1 is written broadly enough that it could apply to any neural network pipeline that separates scene-wide from location-specific features in a driving context, which means it covers a wide slice of the techniques researchers are actively publishing. For anyone tracking new Big Tech patents in the autonomous-vehicle simulation space, this filing is one of the clearer attempts to plant a legal stake around the scene-decomposition approach that has become central to modern AI world models.
That makes this Nvidia's 46th filing we've tracked since May in the self-driving sensing race, following its three-layer camera patent and failsafe car features filing.
Claim 1 as written is quite broad. It does not specify a particular neural network architecture, a particular way of separating the two types of information, or any constraint on what counts as 'spatially-independent.' That breadth is deliberate, and it is also the claim's biggest vulnerability at examination: the USPTO will look for prior art in the large body of published research on disentangled image representations, which goes back years before this filing.
If the claim survives in anything close to its current form, it would cover a meaningful share of the techniques competing teams use to build driving world models. That makes it strategically important for Nvidia, even if the underlying idea, separating global from local image features, is not new to the field.
The practical effect for rivals would depend entirely on how the claim gets narrowed during prosecution. A tighter claim limited to specific architectures or training methods would be easier to design around. In its current form, though, this is a claim worth watching closely as it moves through examination.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
58 drawing sheets from US 2026/0252876 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →