Get the newsletter
Waymo (Google) · Automotive · Filed Mar 30, 2026 · USPTO published Oct 1, 2026 · Oct 1 release

Waymo Patents an AI That Predicts What a Self-Driving Car Will See Next

Waymo is training its self-driving AI not just to react to the world around it, but to mentally simulate what that world is about to do. The patent describes a single neural network that learns to predict the future across every sensor on the car, then gets sharpened for specific driving tasks.

A self-driving car connects to an on-board system for perception and planning, and an off-board system for training the AI model. Drawing from patent filing US 2026/0300725 A1.
A self-driving car connects to an on-board system for perception and planning, and an off-board system for training the AI model.All 11 drawings ↓
Publication number
US 2026/0300725 A1(US20260300725A1)
Applicant
Waymo LLC
Filed · Published
Mar 30, 2026 · Oct 1, 2026
Inventors
Chiyu Jiang, Harish Chandran
Status when we published
Waiting for an examiner
Document
20 claims
US classification
706/25
Parent application
Claims priority from a provisional application 63780046 (filed Mar 28, 2025)

How Waymo's self-driving AI learns to expect the unexpected

What does a self-driving car actually need to stay safe? Reacting to what's in front of it right now is table stakes. The harder part is anticipating what a pedestrian, a truck, or a traffic light is about to do, a split second before it happens.

Waymo's patent covers an AI system designed to build that kind of anticipation. It trains a single neural network on a massive amount of real driving footage, teaching it to predict the next moment in a scene from everything the car's sensors currently see. Cameras, lidar, radar, map data, all of it feeds in together as a stream of data chunks the system calls tokens. The network learns to guess what those tokens look like one moment later.

Once that general-purpose "world model" is trained, Waymo then takes it and fine-tunes separate copies for specific jobs: detecting other vehicles, flagging pedestrian paths, planning the car's next move. Each copy inherits all that prediction ability, then gets specialized. Think of it like training a chef in every cuisine, then assigning them to one kitchen.

From the filing · CLAIM 1
pre-training a multi-modal token processing neural network on a next frame prediction task that requires processing input token sequences characterizing current states of example driving environments to generate output token sequences characterizing predicted future states of the example driving environments; …

Translation: The system first learns by practicing how to guess what happens in the next video frame of traffic.

How the neural network trains across sensors and tasks

The patent describes a two-phase training process for a multi-modal token processing neural network, which is a neural network that can handle multiple types of data at once (cameras, lidar point clouds, radar returns, high-definition maps).

Phase one: pre-training. The network is given sequences of sensor data from real driving scenarios and asked to predict what the next frame of that data looks like. This is sometimes called a "world model" because the system builds an internal representation of how the physical world behaves over time. It never explicitly gets told rules about cars or pedestrians. It learns them by observing.

Phase two: fine-tuning. After pre-training, Waymo creates separate, specialized copies of the network for each specific driving task:

  • Object detection (what's around the car?)
  • Motion prediction (where will those objects go?)
  • Route planning (what should the car do next?)

Each copy gets trained on task-specific data. Because they all start from the same strong foundation, each needs far less additional training than a network built from scratch.

All data types are converted into a unified token format (think of tokens as standardized data units, the same concept used in text AI models), so the network can process everything in one pass rather than running separate pipelines for each sensor.

From the filing · THE ABSTRACT
… obtaining an input sequence of tokens characterizing a driving environment for a vehicle, wherein the input sequence of tokens comprises, for each of one or more time steps, a respective set of tokens for each of a plurality of modalities of data from a set of multiple modalities of data; …

Translation: The car gathers data from multiple types of sensors over time to understand its surroundings.

What this means for self-driving reliability on real roads

The core problem in self-driving is that the real world is full of situations no engineer explicitly programmed for. A system that has learned to predict the physical world rather than just classify objects in it has a fundamentally better shot at handling surprises. That is a meaningful difference in safety terms.

For everyday riders, this architecture also points toward a self-driving stack that improves faster. Because one base model handles all sensor types and then gets specialized per task, Waymo could update the foundation for everything at once rather than maintaining a zoo of separate systems. That could translate to more consistent behavior and fewer edge-case failures, which is where most autonomous vehicle accidents actually originate.

Google's 59th filing we've tracked since May in our self-driving sensing watchlist follows its work on predicting nearby drivers' moves and steering clear of icy roads.

Source: US 2026/0300725 A1 at the USPTO (PDF). Full patent text and figures are in the official publication.

Get our take in your Top Stories

Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Google patent news breaks.

There are more where this came from

We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.

The drawings · 11 sheets

11 drawing sheets from US 2026/0300725 A1 · click any drawing to enlarge

Self-driving car system overview, drawing sheet 1 from US 2026/0300725 A1
Sheet 1 Self-driving car system overview
Self-driving car communication system, drawing sheet 2 from US 2026/0300725 A1
Sheet 2 Self-driving car communication system
Multi-modal token processing network, drawing sheet 3 from US 2026/0300725 A1
Sheet 3 Multi-modal token processing network
Multi-modal network tasks, drawing sheet 4 from US 2026/0300725 A1
Sheet 4 Multi-modal network tasks
Next frame prediction task, drawing sheet 5 from US 2026/0300725 A1
Sheet 5 Next frame prediction task
Tokenization task overview, drawing sheet 6 from US 2026/0300725 A1
Sheet 6 Tokenization task overview
Multi-modal network pre-training, drawing sheet 7 from US 2026/0300725 A1
Sheet 7 Multi-modal network pre-training
Multi-modal network training, drawing sheet 8 from US 2026/0300725 A1
Sheet 8 Multi-modal network training
Show the other 3 sheets
Waymo Patents an AI That Predicts What a Self-Driving Car Will See Next patent drawing, sheet 9
Sheet 9
Waymo Patents an AI That Predicts What a Self-Driving Car Will See Next patent drawing, sheet 10
Sheet 10
Waymo Patents an AI That Predicts What a Self-Driving Car Will See Next patent drawing, sheet 11
Sheet 11
Patent filing page

Watchlists this filing belongs to

All watchlists →
Reader comments

No comments yet

Start the discussion

Real name or a handle, either is fine. Comments are read by a person before they appear, so allow a little time. Keep it about the filing.