Google Files Patent for Shrinking the Example Lists AI Models Reread Every Time
Every time you hand an AI chatbot a stack of examples, it has to reread the whole stack for each new question. Google's patent application describes reading the stack once, saving a compact summary, and reusing that instead.

- Publication number
- US 2026/0368561 A1(US20260368561A1)
- Applicant
- Google LLC
- Filed · Published
- May 15, 2026 · Nov 19, 2026
- Inventors
- Fnu Devvrit, Inderjit Singh Dhillon, Varun Yerram, Nilesh Gupta, Cho-Jui Hsieh, Alec Michael Go, Prateek Jain, Liqian Peng, Jay Yagnik, Venkata Aditya Kusupati, Praneeth Kumar Netrapalli
- Status when we published
- Waiting for an examiner
- Document
- 20 claims
- US classification
- 706/52
What Google's example-compression idea actually does
Imagine you hire an assistant and, before every single question you ask, you make them reread a thick binder of sample answers so they know your style. It works, but it is slow, and the binder only gets thicker as you add more samples.
That is roughly how AI chatbots handle the examples you put in a prompt. Google's filing describes boiling the binder down just once into a short cheat sheet, saving it, and handing the assistant only that sheet whenever a question comes in.
One AI model reads the examples, and another squeezes what it learned into a small package that stays the same size. Later, when you ask something new, the AI checks the package instead of the whole pile. That means less memory used and, in theory, quicker answers.
… processing, by the computing system, the encoded representation and the set of memory tokens with a second machine-learned model to generate a compressed representation of the plurality of exemplars; wherein the compressed representation has a reduced datasize compared to the set of prefix tokens; …
Translation: An AI shrinks down a list of examples so it takes up less space in memory.
How three AI models shrink a pile of examples
The patent covers a way to turn a long list of examples (the filing calls them exemplars) into a much smaller stored summary. Giving a model worked examples inside the prompt is called in-context learning: the model picks up the pattern from the examples without being retrained.
Here is the pipeline described in the claims:
- The examples are broken into prefix tokens (small chunks of text, images or audio that a model reads).
- A first model reads those tokens and produces an encoded representation, a numeric digest of what the examples say.
- A second model combines that digest with a set of memory tokens (a small, fixed-size set of slots the summary gets squeezed into) using cross-attention, a step where one set of data picks out the most relevant parts of another. The output is the compressed representation.
- That result is stored, for example in a cache, and a third model uses it as a conditioning signal (background guidance) when it processes a new question.
Because the memory tokens stay a fixed size, the summary stays the same size no matter how many examples go in. The filing also describes using text, images or audio, using the same model or two different models for the first two steps, and tuning the pieces over repeated rounds of training.
… wherein the compressed representation has been generated by performing cross-attention in a first machine-learned model on a set of prefix tokens or an encoded representation thereof relative to a set of memory tokens or an intermediate-processing representation thereof; …
Translation: The system squeezes the data by comparing the examples against a special set of memory tokens.
Why shorter prompts could mean cheaper, faster AI
If you have ever pasted a long set of samples into a chatbot, that cost shows up as slower replies and, for companies, bigger computing bills. The filing says rereading many examples on every request eats processor time and memory, especially on devices with limited hardware. A compact summary prepared ahead of time could cut that cost and make it practical to give a model many more examples.
The idea is described for text, images and audio alike. For you, the payoff would be faster answers and AI features that could run on smaller devices. For Google, it means serving more requests with the same computers. The filing reports no test numbers, so how much is saved is still unproven.
This is the 97th Google filing we've tracked in our AI assistant and agent coverage since May, building on ideas like loading only needed data and specialist helpers behind one chat box.
Source: US 2026/0368561 A1 at the USPTO (PDF). Full patent text and figures are in the official publication.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Google patent news breaks.
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings · 14 sheets
14 drawing sheets from US 2026/0368561 A1 · click any drawing to enlarge
Show the other 6 sheets

No comments yet