Zoox Patents an AI That Matches Driving Behavior to Plain-English Descriptions
Teaching a self-driving car what 'erratic driver' means is surprisingly hard. Zoox is working on an AI system that learns to connect real vehicle movement patterns to human-written descriptions of that behavior.
How Zoox links real driving moves to plain-English labels
A car cuts across three lanes without signaling. A pedestrian sprints across an intersection mid-block. A cyclist weaves between parked cars. Humans can describe all of these behaviors in a sentence, but until now, computers have had a hard time connecting those descriptions to the actual movement patterns they represent.
Zoox's patent describes a training system that feeds an AI two things at once: real recorded movement paths (called trajectories) from vehicles and people in traffic, and plain-language descriptions of what those movements looked like. By teaching the AI to match descriptions to movements, the system builds a shared understanding of driving behavior across both formats.
Once that foundation is in place, the same AI can be adapted for other tasks, including predicting where a nearby car is going next. That prediction step is critical for any autonomous vehicle deciding when to brake, turn, or hold steady.
receiving input data including at least one of: trajectory data associated with a first object in an environment, or a text description associated with the first object in the environment; and determining, using a trained encoder and based at least in part on the input data, a first encoding …
Translation: The system takes in movement data or text descriptions about an object and converts them into a shared digital format.
How the trajectory and text encoders train against each other
The core idea here is contrastive learning (a training technique where you teach a model what things belong together by also showing it things that don't). In this case, the two things being paired are trajectory data (a recorded sequence of positions and movements for a car, pedestrian, or cyclist) and text descriptions (a sentence or phrase describing that behavior, like 'vehicle accelerating sharply from a stop').
Zoox trains two separate neural network components in tandem:
- A trajectory encoder, which reads movement data and converts it into a compact numerical fingerprint
- A text encoder, which reads written descriptions and converts those into their own fingerprint
The training process pushes matching pairs (same behavior, described two ways) to produce similar fingerprints, while pushing non-matching pairs apart. This is the 'contrastive loss' the patent references, a mathematical penalty that improves with better matching.
Once trained, these encoders act as a reusable foundation. Other models can be bolted on top for specific jobs, most motion forecasting (predicting where nearby objects will move in the next few seconds), which is one of the hardest problems in autonomous driving.
A training system may receive sets of ground truth trajectory data describing movements of objects within driving environments, and associated text descriptions related to the trajectory data.
Translation: The AI learns by looking at real world driving paths paired with matching written descriptions of what happened.
What this means for self-driving car prediction systems
Predicting what other drivers, cyclists, and pedestrians will do next is central to safe autonomous driving. Today, most prediction models are trained on movement data alone. By also training on language descriptions of behavior, Zoox is giving its system a richer way to categorize and recognize what it sees. A model that understands 'hesitant merge' as a concept, not just a shape on a graph, should generalize better to situations it hasn't seen before.
The broader payoff is flexibility. Because the encoders are trained first and then reused, Zoox's steady investment in modular AI training could speed up building new safety features without starting from scratch each time. For riders in an autonomous vehicle, that eventually translates to a car that reads traffic situations more the way an experienced human driver does.
That makes this Amazon's 23rd filing we've tracked since May in our self-driving sensing race, a series that includes Zoox work on remote hand-gesture steering and on-the-fly road mapping.
On the ship-path question, this filing sits in early-to-middle territory. The system described is a training framework, not a deployable feature. Before this touches a real vehicle, Zoox needs large volumes of labeled trajectory data paired with human-written descriptions, and the quality of those text labels matters enormously.
That said, the architecture is software-only. There's no new sensor hardware required, no special chip, and no change to the vehicle itself. The shortest route to a product is integrating the pre-trained encoders into an existing motion forecasting pipeline, which Zoox's patent explicitly names as the target use case.
The real question is data quality and scale. Contrastive learning works well when the paired examples are accurate and varied. If the text descriptions are vague or inconsistent, the encoders learn vague or inconsistent representations. This is a meaningful technical bet, and the approach is sound, but it hinges on getting the training data right before the driving gets better.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
8 drawing sheets from US 2026/0264700 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →