Get the newsletter
Google · AI / ML · Filed May 28, 2025 · USPTO published Nov 19, 2026 · This week's release

Google Files Patent for Speeding Up Chatbots With a Small AI Sidekick

Google describes a way to make AI chatbots answer two to three times faster by letting a tiny model draft the words and a huge one check them.

Sample sentences showing how a quick assistant AI suggests upcoming words so a main AI can write answers faster. Drawing from patent filing US 2026/0368557 A1.
Sample sentences showing how a quick assistant AI suggests upcoming words so a main AI can write answers faster.All 14 drawings ↓
Publication number
US 2026/0368557 A1(US20260368557A1)
Applicant
Google LLC
Filed · Published
May 28, 2025 · Nov 19, 2026
Inventors
Yaniv Leviathan, Matan Kalman, Yossi Matias
Status when we published
Waiting for an examiner
Document
20 claims
US classification
706/52
Parent application
National Stage Entry of PCT/US2023/081347 (filed Nov 28, 2023)(PCTUS2023081347)

What Google's draft-and-check AI trick does

A chatbot writes its answer one word at a time, and every word means running a huge AI model from scratch. That is why you sometimes sit watching a blinking cursor.

Google's patent filing describes a shortcut. A much smaller, cheaper AI guesses several words ahead, like a sous-chef prepping ingredients while the head chef is busy. Then the big model checks all of those guesses in one go, keeping the ones that fit what it would have written and fixing the first one that doesn't.

The filing says the final text comes out statistically the same as if the big model had done every step itself, just faster. In Google's tests, replies arrived two to three times sooner, and the big model did not need to be retrained to get that.

From the filing · CLAIM 1
receiving, by a primary machine learning (ML) model, an input prompt corresponding to a task, wherein the primary ML model is trained to iteratively generate an output token string in response to the input prompt …

Translation: The main AI gets a user prompt and starts generating a response piece by piece.

How the big model grades the small one's guesses

The patent covers a method, a system and a computing device for generating text with two models. A large primary model produces the final answer. A smaller approximating model (a cheaper AI that makes quick guesses) proposes the next piece of text, called a candidate token (a token is a chunk of text, usually a word or part of one).

The steps work like this:

  • The small model looks at the text written so far (the prefix) and samples a guess from its own probability distribution (its list of how likely each possible next word is).
  • The big model, running in parallel, works out its own distribution for that spot and, speculatively, for the spot after the guess (assuming the guess is right).
  • It compares the two. If the small model's odds sit within a set factor of the big model's, the guess is accepted. If not, it is rejected and a replacement is drawn from a corrected distribution built from the difference.

That set factor is a dial the filing calls lenience. Loosening it accepts more guesses and runs faster. In one test pairing an 11 billion parameter T5-XXL with a small helper, settings from strict to loose gave speedups of 2.5x up to 5x.

In an example figure, a 38-token sentence took nine runs of the big model (97 million parameters) with a 6 million parameter helper.

From the filing · THE ABSTRACT
… speculatively determining, based on the candidate token appended to the initial prefix, a target distribution for a next token to be appended to the initial prefix …

Translation: The main model quickly checks if the sidekick's suggested word fits the context.

Why faster chatbot replies could reach you

If you use an AI chatbot, the wait between asking and reading comes largely from this one-word-at-a-time bottleneck. A method that cuts that wait by two to three times, as the filing reports for translation and summarization tests, would make replies feel quicker and could let the same servers handle more people. The filing says the big model's output distribution stays unchanged, so you would get the same quality, only sooner.

The filing also stresses that the helper can be an off-the-shelf small model and that the big model needs no retraining or redesign. That lowers the cost of adopting it. One claim also covers a mobile device, so faster text generation on a phone falls within what the filing describes.

That makes this Google's 40th filing we've tracked since May in our AI teams working together watchlist, which has covered ideas like splitting audio into layers and specialist agents behind one chat box.

Source: US 2026/0368557 A1 at the USPTO (PDF). Full patent text and figures are in the official publication.

Get our take in your Top Stories

Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Google patent news breaks.

There are more where this came from

We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.

The drawings · 14 sheets

14 drawing sheets from US 2026/0368557 A1 · click any drawing to enlarge

A small AI drafts words for a larger model, drawing sheet 1 from US 2026/0368557 A1
Sheet 1 A small AI drafts words for a larger model
Steps for letting a helper AI propose words in batches, drawing sheet 2 from US 2026/0368557 A1
Sheet 2 Steps for letting a helper AI propose words in batches
Word output gains measured across different helper AI sizes, drawing sheet 3 from US 2026/0368557 A1
Sheet 3 Word output gains measured across different helper AI sizes
How often large AI models accept helper guesses, drawing sheet 4 from US 2026/0368557 A1
Sheet 4 How often large AI models accept helper guesses
Calculating the ideal guess count based on helper AI speed, drawing sheet 5 from US 2026/0368557 A1
Sheet 5 Calculating the ideal guess count based on helper AI speed
Comparing computing effort against total chatbot response speed, drawing sheet 6 from US 2026/0368557 A1
Sheet 6 Comparing computing effort against total chatbot response speed
Plotting overall speed boosts against extra computing steps, drawing sheet 7 from US 2026/0368557 A1
Sheet 7 Plotting overall speed boosts against extra computing steps
Quicker replies when a helper AI assists a larger model, drawing sheet 8 from US 2026/0368557 A1
Sheet 8 Quicker replies when a helper AI assists a larger model
Show the other 6 sheets
Measured speedups using smaller AI models to draft answers, drawing sheet 9 from US 2026/0368557 A1
Sheet 9 Measured speedups using smaller AI models to draft answers
How an AI system learns from data before answering requests, drawing sheet 10 from US 2026/0368557 A1
Sheet 10 How an AI system learns from data before answering requests
Internal parts of a device running the AI software, drawing sheet 11 from US 2026/0368557 A1
Sheet 11 Internal parts of a device running the AI software
Everyday gadgets communicating with central servers over the internet, drawing sheet 12 from US 2026/0368557 A1
Sheet 12 Everyday gadgets communicating with central servers over the internet
Cloud servers grouped in clusters to handle large AI workloads, drawing sheet 13 from US 2026/0368557 A1
Sheet 13 Cloud servers grouped in clusters to handle large AI workloads
How the main AI checks words drafted by its helper, drawing sheet 14 from US 2026/0368557 A1
Sheet 14 How the main AI checks words drafted by its helper
Patent filing page

More in AI assistant & agent

All 436 AI assistant & agent patents →

Watchlists this filing belongs to

All watchlists →
Reader comments

No comments yet

Start the discussion

Real name or a handle, either is fine. Comments are read by a person before they appear, so allow a little time. Keep it about the filing.