Google Patents an AI Memory That Stays the Same Size, However Long the Conversation Runs
Chatbots keep a note on every word you've said, and that pile grows without limit. Google's new patent application squeezes it into a memory that stays the same size.
What Google's fixed-size AI memory does
A chatbot is an hour into helping you plan a trip, and it is still hauling around a note for every single word the two of you have exchanged. Each new word adds another note to the pile. The longer you talk, the more computer memory it takes just to keep up.
Google's patent application describes a different habit. Instead of keeping a note for every word, the AI keeps a fixed-size notebook with a set number of slots. As each new word arrives, the AI folds the useful parts into the notebook, then writes its next word by reading from the notebook instead of the whole pile.
The notebook stays the same size however long the conversation runs. The filing says that makes the approach a better fit for phones, smart home gadgets and other devices with little memory to spare.
Systems and methods for generating a network output using a neural network. In particular, the neural network includes one or more compressive attention layers that each “compress” information into a respective fixed size memory state.
Translation: The system uses special neural network layers to shrink conversation data into a constant memory size.
How the memory updates with each new word
Most modern chatbots use attention, a step where the AI looks back over earlier words to decide what matters for the next one. To avoid redoing work, they save numbers for every earlier word in a key-value cache (a scrapbook of past context). That cache gets bigger with every token (a word or word fragment).
The claimed method swaps the growing cache for a memory state with a fixed number of slots. For each new token, the layer produces three lists of numbers called a query, a key and a value. It then runs optimization steps on a compression objective (a score for how well the memory can rebuild the value when given the key) to update the memory. Finally, it applies the updated memory to the query to produce its output.
The filing describes several refinements:
- A single gradient descent step (one small corrective nudge) per token can do the update.
- Each slot takes in only the part of new information it doesn't already hold, which limits overwriting.
- Slots are kept at a standard size so the numbers don't blow up.
- A forget gate can let old information fade.
In the filing's example, called Lattice, models with 110 million parameters reached lower perplexity (how badly a model predicts the next word; lower is better) than the comparison models, across context lengths from 512 to 16,000 tokens.
… performing a set of one or more optimization steps on a compression objective using the key vector and the value vector to update the memory state for the compressive attention layer; and after updating the memory state for the compressive attention layer, applying the memory state to the query vector to generate a layer output token for the compressive attention layer.
Translation: It continuously updates and applies the compressed memory state to generate the next part of the output.
Why a memory cap matters for phones and servers
Memory is the bottleneck here. The fast memory on AI chips is limited and expensive, and a growing cache eats it, which caps how long a conversation can be and how many people one server can handle at once. A fixed-size memory lifts that ceiling. Because the size is known in advance, the filing says the model can also be tuned to use exactly the memory a given device has, such as a phone or a smart home gadget.
For you, that could mean more capable AI running on your own device instead of in a distant data center. Keep the caveats in view: this is a patent application, and the comparisons in it cover small models only.
Google's 38th filing we've tracked in our AI assistants that remember you watchlist since May builds on earlier applications like one on answer-style memory and one on earbud conversation capture.
Chatbot memory costs a lot, and the bill grows with every word. The fast memory on AI chips is the scarcest and priciest part of the hardware, and a record that expands with each word limits how long a chat can run and how many people one machine can serve.
Google's answer goes after the growth itself by capping the record at a fixed size, which fits the scale of the cost well. A cap forces the model to forget something, and the filing handles that by writing into each slot only what it doesn't already hold.
The evidence is thin so far: tests on models with 110 million parameters, small next to the chatbots you actually use. If the result holds at larger sizes, it is a meaningful cost saver. If it doesn't, it stays a promising lab idea.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Google patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
4 drawing sheets from US 2026/0310945 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in