Get the newsletter
Nvidia · AI / ML · Filed May 14, 2025 · USPTO published Nov 19, 2026 · This week's release

Nvidia Files Patent for Shrinking the Notes AI Chatbots Keep on Your Conversation

Every long AI conversation leaves behind a huge pile of saved numbers that has to be parked and fetched again. Nvidia's patent describes a way to shrink that pile before it moves.

Blocks of saved conversation data are broken down, organized into rows, and squished down to take up less memory on an AI chip. Drawing from patent filing US 2026/0368516 A1.
Blocks of saved conversation data are broken down, organized into rows, and squished down to take up less memory on an AI chip.All 26 drawings ↓
Publication number
US 2026/0368516 A1(US20260368516A1)
Applicant
NVIDIA Corporation
Filed · Published
May 14, 2025 · Nov 19, 2026
Inventors
Przemek Strzelczyk, Przemyslaw Dominik Podczasi, Konrad Stanislaw Staniszewski, Adrian Robert Lancucki
Status when we published
Waiting for an examiner
Document
20 claims
US classification
706/25

What Nvidia's chatbot-memory compression does

Every time you reply to an AI chatbot, it has to pull up everything it has already worked out about your conversation so far. That running memory (the technical name is a KV cache) lives on the graphics chip, and it gets huge when the chat, or the document you pasted in, is long.

To free up room, a service can park that memory on a separate drive while you're thinking, then fetch it back when you hit send. Moving that much data takes time. Nvidia's patent shrinks the memory before it moves: it pulls out the bookkeeping about word order, lines up the remaining numbers so similar values sit together, and then squeezes them.

The steps run in reverse when your reply arrives, in small chunks, so the chatbot can start working before everything has been loaded.

From the filing · CLAIM 1
… divide data stored in the KV cache into Key (K) and Value (V) data according to one or more attention heads of one or more layers of the one or more neural networks; and realign the divided K and V data separately by performing one or more orthogonal projections.

Translation: The system splits chat memory parts and rotates them mathematically to save space.

How the saved numbers get lined up and squeezed

When a chatbot reads your prompt, each layer of the model produces two sets of numbers for every word: keys (labels for what each word is about) and values (the content to pull from). Saved together, they form the KV cache. Several parallel attention heads (small units that each track a different kind of relationship between words) produce their own keys and values.

The patent's claims cover these steps:

  • Store each word's position information separately, outside the KV cache.
  • Split the cache into keys and values, grouped by attention head and layer.
  • Realign keys and values separately with orthogonal projections (a rotation of the data, in the same family as principal component analysis) so that similar values line up.

Dependent claims add the shrinking steps. Quantizing stores numbers with fewer bits (the description gives 8 instead of 32 as an example). Then a lossless or lossy compression method is applied, and the result is written to separate storage such as a drive. For floating-point numbers (the format computers use for fractions), the exponent and mantissa can be compressed by different methods. A small scoring model, such as a linear regression, can check how much accuracy a lossy method costs and switch methods.

Retrieval runs backward: decompress, rotate the data back, recombine by head and layer, add the position information, and store it on the chip. The work happens in chunks, so loading can overlap with use.

From the filing · THE ABSTRACT
Apparatuses, systems, and techniques to compress Key Value (KV) data of a neural network are described. In at least one embodiment, position information of context tokens is stored separate from KV cache data and the KV cache data may be divided, realigned, and compressed before storage in an auxiliary storage.

Translation: Nvidia invented a method to shrink chat memory by separating and squeezing the data.

Why long AI chats strain GPU memory today

If you chat with an AI for a long stretch, or hand it a big document, the memory it keeps about that context takes up scarce space on expensive graphics chips. Shrinking that saved memory means less data to move when a conversation is parked and resumed, which the filing says cuts storage use and delay. For you, that could mean quicker replies, or cheaper service for the companies running these systems.

The filing describes chatbots, image-and-text models and other systems that predict one piece at a time, and it describes use in data centers. It names no product. Its pitch is that the saved context can shrink with little or no loss in accuracy, depending on which compression method gets picked.

This is the 68th Nvidia patent we've tracked since July in the AI chip wars, joining earlier applications like one that saves idle power and one giving buyers software control.

Source: US 2026/0368516 A1 at the USPTO (PDF). Full patent text and figures are in the official publication.

Get our take in your Top Stories

Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Nvidia patent news breaks.

There are more where this came from

We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.

The drawings · 26 sheets

26 drawing sheets from US 2026/0368516 A1 · click any drawing to enlarge

An AI system that shrinks stored conversation memory, drawing sheet 1 from US 2026/0368516 A1
Sheet 1 An AI system that shrinks stored conversation memory
Stripping word order data to compress chatbot memory, drawing sheet 2 from US 2026/0368516 A1
Sheet 2 Stripping word order data to compress chatbot memory
Restoring word order data to unpack saved conversation memory, drawing sheet 3 from US 2026/0368516 A1
Sheet 3 Restoring word order data to unpack saved conversation memory
Steps for packing conversation memory into smaller storage, drawing sheet 4 from US 2026/0368516 A1
Sheet 4 Steps for packing conversation memory into smaller storage
Unpacking saved conversation memory when the AI resumes chatting, drawing sheet 5 from US 2026/0368516 A1
Sheet 5 Unpacking saved conversation memory when the AI resumes chatting
Server center equipment that runs large AI models, drawing sheet 6 from US 2026/0368516 A1
Sheet 6 Server center equipment that runs large AI models
All-in-one chip that handles computing and memory tasks, drawing sheet 7 from US 2026/0368516 A1
Sheet 7 All-in-one chip that handles computing and memory tasks
High-speed processor built to run heavy AI calculations, drawing sheet 8 from US 2026/0368516 A1
Sheet 8 High-speed processor built to run heavy AI calculations
Show the other 18 sheets
How graphics processing cores share memory to handle AI tasks, drawing sheet 9 from US 2026/0368516 A1
Sheet 9 How graphics processing cores share memory to handle AI tasks
Inside parts of a chip that speed up AI tasks, drawing sheet 10 from US 2026/0368516 A1
Sheet 10 Inside parts of a chip that speed up AI tasks
Inside a general processor core that runs AI workloads, drawing sheet 11 from US 2026/0368516 A1
Sheet 11 Inside a general processor core that runs AI workloads
Specialized hardware designed to run neural network computations quickly, drawing sheet 12 from US 2026/0368516 A1
Sheet 12 Specialized hardware designed to run neural network computations quickly
Brain-inspired chip circuits that mimic biological neurons to run AI, drawing sheet 13 from US 2026/0368516 A1
Sheet 13 Brain-inspired chip circuits that mimic biological neurons to run AI
Network of supercomputer server racks that process user requests, drawing sheet 14 from US 2026/0368516 A1
Sheet 14 Network of supercomputer server racks that process user requests
Multi-chip processor layout paired with high-speed memory for AI jobs, drawing sheet 15 from US 2026/0368516 A1
Sheet 15 Multi-chip processor layout paired with high-speed memory for AI jobs
Main chip parts linked together to power built-in AI, drawing sheet 16 from US 2026/0368516 A1
Sheet 16 Main chip parts linked together to power built-in AI
Internal parts of a processor core handling AI calculations, drawing sheet 17 from US 2026/0368516 A1
Sheet 17 Internal parts of a processor core handling AI calculations
Specialized circuits designed to speed up heavy AI math, drawing sheet 18 from US 2026/0368516 A1
Sheet 18 Specialized circuits designed to speed up heavy AI math
An alternative chip layout built around open processor standards, drawing sheet 19 from US 2026/0368516 A1
Sheet 19 An alternative chip layout built around open processor standards
Grid layouts linking multiple small processor units together, drawing sheet 20 from US 2026/0368516 A1
Sheet 20 Grid layouts linking multiple small processor units together
Software layers connecting an AI app to physical chips, drawing sheet 21 from US 2026/0368516 A1
Sheet 21 Software layers connecting an AI app to physical chips
Translating written software into instructions a computer can run, drawing sheet 22 from US 2026/0368516 A1
Sheet 22 Translating written software into instructions a computer can run
Converting programs to run across rival brands of AI hardware, drawing sheet 23 from US 2026/0368516 A1
Sheet 23 Converting programs to run across rival brands of AI hardware
Self-driving car components that feed road data into onboard AI, drawing sheet 24 from US 2026/0368516 A1
Sheet 24 Self-driving car components that feed road data into onboard AI
Chip components that calculate and store AI answers, drawing sheet 25 from US 2026/0368516 A1
Sheet 25 Chip components that calculate and store AI answers
How an AI model learns from data to produce answers, drawing sheet 26 from US 2026/0368516 A1
Sheet 26 How an AI model learns from data to produce answers
Patent filing page

More in AI assistant & agent

All 436 AI assistant & agent patents →

Watchlists this filing belongs to

All watchlists →
Reader comments

No comments yet

Start the discussion

Real name or a handle, either is fine. Comments are read by a person before they appear, so allow a little time. Keep it about the filing.