Nvidia Files Patent for Shrinking the Notes AI Chatbots Keep on Your Conversation
Every long AI conversation leaves behind a huge pile of saved numbers that has to be parked and fetched again. Nvidia's patent describes a way to shrink that pile before it moves.

- Publication number
- US 2026/0368516 A1(US20260368516A1)
- Applicant
- NVIDIA Corporation
- Filed · Published
- May 14, 2025 · Nov 19, 2026
- Inventors
- Przemek Strzelczyk, Przemyslaw Dominik Podczasi, Konrad Stanislaw Staniszewski, Adrian Robert Lancucki
- Status when we published
- Waiting for an examiner
- Document
- 20 claims
- US classification
- 706/25
What Nvidia's chatbot-memory compression does
Every time you reply to an AI chatbot, it has to pull up everything it has already worked out about your conversation so far. That running memory (the technical name is a KV cache) lives on the graphics chip, and it gets huge when the chat, or the document you pasted in, is long.
To free up room, a service can park that memory on a separate drive while you're thinking, then fetch it back when you hit send. Moving that much data takes time. Nvidia's patent shrinks the memory before it moves: it pulls out the bookkeeping about word order, lines up the remaining numbers so similar values sit together, and then squeezes them.
The steps run in reverse when your reply arrives, in small chunks, so the chatbot can start working before everything has been loaded.
… divide data stored in the KV cache into Key (K) and Value (V) data according to one or more attention heads of one or more layers of the one or more neural networks; and realign the divided K and V data separately by performing one or more orthogonal projections.
Translation: The system splits chat memory parts and rotates them mathematically to save space.
How the saved numbers get lined up and squeezed
When a chatbot reads your prompt, each layer of the model produces two sets of numbers for every word: keys (labels for what each word is about) and values (the content to pull from). Saved together, they form the KV cache. Several parallel attention heads (small units that each track a different kind of relationship between words) produce their own keys and values.
The patent's claims cover these steps:
- Store each word's position information separately, outside the KV cache.
- Split the cache into keys and values, grouped by attention head and layer.
- Realign keys and values separately with orthogonal projections (a rotation of the data, in the same family as principal component analysis) so that similar values line up.
Dependent claims add the shrinking steps. Quantizing stores numbers with fewer bits (the description gives 8 instead of 32 as an example). Then a lossless or lossy compression method is applied, and the result is written to separate storage such as a drive. For floating-point numbers (the format computers use for fractions), the exponent and mantissa can be compressed by different methods. A small scoring model, such as a linear regression, can check how much accuracy a lossy method costs and switch methods.
Retrieval runs backward: decompress, rotate the data back, recombine by head and layer, add the position information, and store it on the chip. The work happens in chunks, so loading can overlap with use.
Apparatuses, systems, and techniques to compress Key Value (KV) data of a neural network are described. In at least one embodiment, position information of context tokens is stored separate from KV cache data and the KV cache data may be divided, realigned, and compressed before storage in an auxiliary storage.
Translation: Nvidia invented a method to shrink chat memory by separating and squeezing the data.
Why long AI chats strain GPU memory today
If you chat with an AI for a long stretch, or hand it a big document, the memory it keeps about that context takes up scarce space on expensive graphics chips. Shrinking that saved memory means less data to move when a conversation is parked and resumed, which the filing says cuts storage use and delay. For you, that could mean quicker replies, or cheaper service for the companies running these systems.
The filing describes chatbots, image-and-text models and other systems that predict one piece at a time, and it describes use in data centers. It names no product. Its pitch is that the saved context can shrink with little or no loss in accuracy, depending on which compression method gets picked.
This is the 68th Nvidia patent we've tracked since July in the AI chip wars, joining earlier applications like one that saves idle power and one giving buyers software control.
Source: US 2026/0368516 A1 at the USPTO (PDF). Full patent text and figures are in the official publication.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Nvidia patent news breaks.
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings · 26 sheets
26 drawing sheets from US 2026/0368516 A1 · click any drawing to enlarge
Show the other 18 sheets

No comments yet