Google Files Patent for Speeding Up Chatbots With a Small AI Sidekick
Google describes a way to make AI chatbots answer two to three times faster by letting a tiny model draft the words and a huge one check them.

- Publication number
- US 2026/0368557 A1(US20260368557A1)
- Applicant
- Google LLC
- Filed · Published
- May 28, 2025 · Nov 19, 2026
- Inventors
- Yaniv Leviathan, Matan Kalman, Yossi Matias
- Status when we published
- Waiting for an examiner
- Document
- 20 claims
- US classification
- 706/52
- Parent application
- National Stage Entry of PCT/US2023/081347 (filed Nov 28, 2023)(PCTUS2023081347)
What Google's draft-and-check AI trick does
A chatbot writes its answer one word at a time, and every word means running a huge AI model from scratch. That is why you sometimes sit watching a blinking cursor.
Google's patent filing describes a shortcut. A much smaller, cheaper AI guesses several words ahead, like a sous-chef prepping ingredients while the head chef is busy. Then the big model checks all of those guesses in one go, keeping the ones that fit what it would have written and fixing the first one that doesn't.
The filing says the final text comes out statistically the same as if the big model had done every step itself, just faster. In Google's tests, replies arrived two to three times sooner, and the big model did not need to be retrained to get that.
receiving, by a primary machine learning (ML) model, an input prompt corresponding to a task, wherein the primary ML model is trained to iteratively generate an output token string in response to the input prompt …
Translation: The main AI gets a user prompt and starts generating a response piece by piece.
How the big model grades the small one's guesses
The patent covers a method, a system and a computing device for generating text with two models. A large primary model produces the final answer. A smaller approximating model (a cheaper AI that makes quick guesses) proposes the next piece of text, called a candidate token (a token is a chunk of text, usually a word or part of one).
The steps work like this:
- The small model looks at the text written so far (the prefix) and samples a guess from its own probability distribution (its list of how likely each possible next word is).
- The big model, running in parallel, works out its own distribution for that spot and, speculatively, for the spot after the guess (assuming the guess is right).
- It compares the two. If the small model's odds sit within a set factor of the big model's, the guess is accepted. If not, it is rejected and a replacement is drawn from a corrected distribution built from the difference.
That set factor is a dial the filing calls lenience. Loosening it accepts more guesses and runs faster. In one test pairing an 11 billion parameter T5-XXL with a small helper, settings from strict to loose gave speedups of 2.5x up to 5x.
In an example figure, a 38-token sentence took nine runs of the big model (97 million parameters) with a 6 million parameter helper.
… speculatively determining, based on the candidate token appended to the initial prefix, a target distribution for a next token to be appended to the initial prefix …
Translation: The main model quickly checks if the sidekick's suggested word fits the context.
Why faster chatbot replies could reach you
If you use an AI chatbot, the wait between asking and reading comes largely from this one-word-at-a-time bottleneck. A method that cuts that wait by two to three times, as the filing reports for translation and summarization tests, would make replies feel quicker and could let the same servers handle more people. The filing says the big model's output distribution stays unchanged, so you would get the same quality, only sooner.
The filing also stresses that the helper can be an off-the-shelf small model and that the big model needs no retraining or redesign. That lowers the cost of adopting it. One claim also covers a mobile device, so faster text generation on a phone falls within what the filing describes.
That makes this Google's 40th filing we've tracked since May in our AI teams working together watchlist, which has covered ideas like splitting audio into layers and specialist agents behind one chat box.
Source: US 2026/0368557 A1 at the USPTO (PDF). Full patent text and figures are in the official publication.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Google patent news breaks.
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings · 14 sheets
14 drawing sheets from US 2026/0368557 A1 · click any drawing to enlarge
Show the other 6 sheets

No comments yet