Waymo Patents an AI That Predicts What a Self-Driving Car Will See Next
Waymo is training its self-driving AI not just to react to the world around it, but to mentally simulate what that world is about to do. The patent describes a single neural network that learns to predict the future across every sensor on the car, then gets sharpened for specific driving tasks.
How Waymo's self-driving AI learns to expect the unexpected
What does a self-driving car actually need to stay safe? Reacting to what's in front of it right now is table stakes. The harder part is anticipating what a pedestrian, a truck, or a traffic light is about to do, a split second before it happens.
Waymo's patent covers an AI system designed to build that kind of anticipation. It trains a single neural network on a massive amount of real driving footage, teaching it to predict the next moment in a scene from everything the car's sensors currently see. Cameras, lidar, radar, map data, all of it feeds in together as a stream of data chunks the system calls tokens. The network learns to guess what those tokens look like one moment later.
Once that general-purpose "world model" is trained, Waymo then takes it and fine-tunes separate copies for specific jobs: detecting other vehicles, flagging pedestrian paths, planning the car's next move. Each copy inherits all that prediction ability, then gets specialized. Think of it like training a chef in every cuisine, then assigning them to one kitchen.
pre-training a multi-modal token processing neural network on a next frame prediction task that requires processing input token sequences characterizing current states of example driving environments to generate output token sequences characterizing predicted future states of the example driving environments; …
Translation: The system first learns by practicing how to guess what happens in the next video frame of traffic.
How the neural network trains across sensors and tasks
The patent describes a two-phase training process for a multi-modal token processing neural network, which is a neural network that can handle multiple types of data at once (cameras, lidar point clouds, radar returns, high-definition maps).
Phase one: pre-training. The network is given sequences of sensor data from real driving scenarios and asked to predict what the next frame of that data looks like. This is sometimes called a "world model" because the system builds an internal representation of how the physical world behaves over time. It never explicitly gets told rules about cars or pedestrians. It learns them by observing.
Phase two: fine-tuning. After pre-training, Waymo creates separate, specialized copies of the network for each specific driving task:
- Object detection (what's around the car?)
- Motion prediction (where will those objects go?)
- Route planning (what should the car do next?)
Each copy gets trained on task-specific data. Because they all start from the same strong foundation, each needs far less additional training than a network built from scratch.
All data types are converted into a unified token format (think of tokens as standardized data units, the same concept used in text AI models), so the network can process everything in one pass rather than running separate pipelines for each sensor.
… obtaining an input sequence of tokens characterizing a driving environment for a vehicle, wherein the input sequence of tokens comprises, for each of one or more time steps, a respective set of tokens for each of a plurality of modalities of data from a set of multiple modalities of data; …
Translation: The car gathers data from multiple types of sensors over time to understand its surroundings.
What this means for self-driving reliability on real roads
The core problem in self-driving is that the real world is full of situations no engineer explicitly programmed for. A system that has learned to predict the physical world rather than just classify objects in it has a fundamentally better shot at handling surprises. That is a meaningful difference in safety terms.
For everyday riders, this architecture also points toward a self-driving stack that improves faster. Because one base model handles all sensor types and then gets specialized per task, Waymo could update the foundation for everything at once rather than maintaining a zoo of separate systems. That could translate to more consistent behavior and fewer edge-case failures, which is where most autonomous vehicle accidents actually originate.
Google's 59th filing we've tracked since May in our self-driving sensing watchlist follows its work on predicting nearby drivers' moves and steering clear of icy roads.
The problem this patent takes on is as real as it gets in autonomous driving. Fully self-driving cars still struggle most with rare, unpredictable situations: a cyclist cutting across lanes, a ball rolling into the street, a truck reversing in an intersection. Rule-based systems can't cover every case. Prediction-based AI that has absorbed millions of real driving moments is at least theoretically more prepared.
The approach here, training a single general model to predict the future, then cloning and specializing it, matches the scale of that problem. It mirrors what made large language models effective: pre-train on everything, fine-tune for the specific job. Applying that playbook to physical-world sensor data is a logical and serious move, not a quirky experiment.
Waymo's track record in world-model patents gives this filing more credibility than it would have from a newcomer. The question worth asking is not whether the architecture is sound, but whether the real-world driving data Waymo feeds into it is varied and large enough to matter. That is an operational question the patent can't answer by itself.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
11 drawing sheets from US 2026/0300725 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in