New Google Patent Predicts How People and Vehicles Move Through Real-World Scenes

By Patentlyze Team · Updated Jul 27, 2026

Google is patenting a neural network system that watches a scene full of moving objects, fills in what it can't directly see, and predicts where everything is headed next. That's a core capability for self-driving cars and robotics.

See all 4 drawings from this filing ↓

How Google's behavior-prediction system actually works

Imagine a self-driving car trying to figure out what every pedestrian, cyclist, and other car on the road is about to do. It can't watch all of them perfectly at every moment, so it has to make educated guesses about the ones it partially sees or briefly loses track of.

This Google patent describes a system that takes a snapshot of a scene across multiple moments in time, treats missing or partially hidden objects as blanks to fill in, and then predicts future behavior for everything in the scene at once. Think of it like a very structured game of "fill in the blank" played across both space and time.

The system uses two neural networks working in sequence: one to compress and understand the scene, and a second to generate predictions about what comes next. It's designed to handle multiple agents (any moving thing) simultaneously, which is exactly what you need when real-world environments are crowded and chaotic.

Inside Google's encoder-decoder behavior model

The patent describes a pipeline built around two neural networks working in sequence, an encoder and a decoder.

The input to the system is a 3D tensor (think of it as a three-dimensional data grid, where the axes represent agents, time steps, and descriptive features). Crucially, some cells in that grid are masked, meaning they represent things the system didn't observe directly. This is a deliberate design choice borrowed from a technique called masked autoencoding, which trains models to be good at reconstruction by forcing them to work with incomplete information.

• The encoder network processes the full grid (observed and masked cells together) into a compressed internal representation.
• The decoder network then takes that compressed representation and produces a 4D output tensor (adding a fourth axis, likely for predicted future time steps or probability distributions over possible futures).
• The system is designed to handle multiple tasks, including trajectory prediction and behavior classification, from the same learned model.

The claim that was active at publication has since been canceled, which is common during patent prosecution and does not necessarily indicate the application is abandoned. The core technical framing, a masked 3D-to-4D encoder-decoder for multi-agent behavior, is the intellectual territory Google is staking out.

What this means for self-driving and crowd AI

Predicting what multiple moving objects will do next is one of the hardest problems in autonomous systems. A self-driving car that can only track one pedestrian at a time, or that falls apart when sensor data goes briefly missing, is not safe enough to deploy at scale. A system that handles crowds of agents across time with built-in tolerance for incomplete observations is a meaningful step toward the reliability those systems need.

Google's Waymo division is one of the most active players in autonomous driving, and this kind of multi-agent behavior modeling sits at the heart of how those vehicles understand the world around them. Whether or not this specific patent directly maps to a Waymo system, it reflects where the research is pointed.

The drawings

4 drawing sheets from US 2026/0212162 A1 · click any drawing to enlarge

SHEET 1 SHEET 2 SHEET 3 SHEET 4

Browse all the drawings 4 drawing sheets

× ← →

Editorial commentary on a publicly published patent application. Not legal advice.