A New Nvidia Patent Spreads a Chatbot's Long Memory Across Many Chips
Ask a chatbot to read a million words and it has to keep flipping back through its own notes. Nvidia's new patent application describes a way to spread that flipping across many chips.
What Nvidia's chip-sharing chatbot patent does
Imagine you paste an entire novel into a chatbot and ask it to summarize chapter twelve. Before it answers, the chatbot has to look back through everything you gave it, again and again, for every new word it writes. All that looking back eats up memory and slows the reply.
Nvidia's filing describes a way to share that work among several chips (the processors that run AI) instead of making one chip carry the whole load. Each chip keeps just a slice of the chatbot's notes about your text and does its part of the reading. Then the chips swap their results.
The second half of the job, where the chatbot polishes its answer, gets divided up in a different pattern. That keeps every chip busy instead of leaving some waiting around, and the aim is quicker replies even when the text you hand over is enormous.
… Key-Value (KV) attention heads and partitions of a KV cache of context data of the transformer are distributed to different ones of the plurality of circuits …
Translation: The chatbot memory is split up across multiple processing chips instead of living on just one.
How the attention and answer steps get split
Most modern chatbots run on a design called a transformer. It has two main stages: attention, which works out which earlier words matter for the next one, and a feed-forward network (FFN), which expands and enriches what attention found. To avoid redoing work, the system stores earlier results in a KV cache (a running notebook of how the words in the context relate to each other). With contexts that run from tens of thousands to millions of tokens, loading that notebook becomes a bottleneck.
The patent splits the work two different ways:
- Attention stage: the attention heads (separate readers, each tracking a different kind of relationship) and slices of the KV cache are spread across different processors. Each one computes its share.
- Swap: the processors exchange their partial attention results using a reduce scatter (each chip combines everyone's numbers, then hands each peer its portion).
- FFN stage: the FFN's weights (the learned numbers it relies on) are cut into more pieces than the attention stage used. In the filing's example, attention is split two ways while the FFN is split four ways.
The reason is a memory-saving design called Grouped-Query Attention (or its extreme form, Multi-Query Attention), where many query heads share a few key-value heads. For instance, 16 heads might share 4 groups. Splitting attention across more chips than there are groups would force copies of the cache, wasting the savings. The FFN holds roughly two-thirds of a model's parameters, so spreading it wider cuts how much each chip must load. The claims cover this as a processor, a system, and a method, with GPUs as the example chips.
The attention values may be exchanged between the different processors. An output of the transformer may be computed at the different processors using different weight partitions of a Feedforward Network …
Translation: The chips constantly share their notes so the AI can figure out what to say next.
What this means for long-context AI on Nvidia GPUs
If you have ever pasted a long report or a big pile of code into an AI assistant and waited, part of that wait comes from chips shuffling the assistant's memory around. The filing describes a way to cut that shuffling when the input runs from tens of thousands to millions of tokens (tokens are the word-sized pieces the AI reads). It also targets cases where only a few requests arrive at once and replies need to come back fast.
The filing names uses like summarizing long documents, analyzing large codebases, and multi-agent dialogue. It also places the system in data centers, which is where Nvidia's GPUs do most of this work. Better use of each chip would let a company serve more answers from the same hardware, though the patent gives no speed numbers.
Nvidia's 61st filing we've tracked since July in the AI chip wars follows one about regrouping chips mid-reply and one about gathering and sharing partial results.
Claim 1 covers any system that runs a neural network with transformers by splitting the memory of past conversation across multiple chips, exchanging results between those chips, and then splitting the second round of computation differently from the first. It names no specific number of chips, no required method of exchange, and no particular ratio for either split.
That scope reaches the infrastructure behind nearly every AI service handling long conversations at scale, from customer support tools to document summarizers, because dividing the workload in exactly these two stages is a standard way to handle the memory demands of extended back-and-forth dialogue efficiently.
If granted as written, Claim 1 would give Nvidia a hand on the steering wheel of how competing hardware makers design multi-chip servers for large language models, since almost any sensible architecture for the problem falls within what the claim describes.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Nvidia patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
26 drawing sheets from US 2026/0350353 A1 · click any drawing to enlarge
Show the other 18 sheets
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in