A New Nvidia Patent Reworks How Many Chips Share One Chatbot's Workload
Behind every chatbot answer, dozens of chips have to coordinate, and the chatter between them costs time. Nvidia's filing describes a way to cut that chatter with one simple swap, then put the same chips to work on a different job.
What Nvidia's chip-sharing chatbot plan does
Ever been stuck in a group project where everyone finishes their part, then spends forever swapping notes before the next step? The computers that serve AI chatbots run into a version of that problem.
When thousands of people ask a chatbot questions at once, a data center spreads those requests across many chips. Those chips have to store each conversation's running memory and keep reloading the model's enormous set of learned numbers. Both jobs eat time, and passing results between chips eats more.
Nvidia's filing describes a way to divide the work in two stages. First, each chip handles its own share of the conversations. Then the chips make one simple swap of partial results, and each chip takes a different slice of the model's next stage. In the filing's worked example, that swap sends roughly 78% less data than a common alternative.
receives a batch partition of the batch of input data to compute attention values for a Key-Value (KV) attention head partition that uses a KV cache of context data for the inferences …
Translation: The chip handles a specific slice of the workload using stored conversation history.
How the chips trade notes mid-calculation
A transformer (the design behind most chatbots) works in two stages. The first is attention, where the model works out which earlier words matter for the next one. The second is a feed-forward network, or FFN (a big block of learned numbers that enriches each word's representation). The patent covers running a batch of requests across several processors, such as GPUs, in a way that suits both stages.
In the first stage, each processor receives a slice of the batch and a slice of the KV attention heads (separate "viewpoints" the model uses to relate words). Each one also reads its matching piece of the KV cache, a stored record of earlier context so the model does not recompute it.
- Split the batch and the attention heads across processors.
- Compute attention values locally on each processor.
- Swap results with an All-to-All exchange, where every processor sends data to every other one. No adding is involved, so lower-precision number formats such as 4-bit floating point can be used.
- Compute the output, with each processor holding a different slice of the FFN weights.
The filing works through one example: a batch of 64, a hidden size of 8192, and 8 processors. The swap moves about 112 KB, against 512 KB for an All-Gather approach. Memory reads per processor drop from about 64 MB to about 8 MB. The claims also cover grouping GPUs into ranks and assigning query heads.
… exchanges two or more of the attention values between the different processors, and computes an output of the transformer at different ones of the processors using different weight partitions of a Feed-forward Network (FFN).
Translation: Multiple chips share intermediate results and split up the final math.
Why faster chatbot replies start with chips
If you use a chatbot, the payoff is speed. The slow part of answering is often chips waiting on data, either from each other or from memory. Trimming the data they swap, and reloading less of the model per chip, targets exactly that wait. The filing says the approach suits small to medium conversation lengths, meaning hundreds to thousands of tokens (word pieces).
For Nvidia, this is software-and-hardware know-how that makes its chips look better at serving AI models. The filing says it is not tied to one product: it lists GPUs, CPUs, data centers and many other processor types as places the method could run. Anyone running a large chatbot service cares about response time, so even modest gains in how chips divide the work could matter at that scale.
Nvidia's 62nd filing we've tracked since July in our AI chip wars watch builds on spreading memory across chips and regrouping chips mid-reply.
You will never see this feature. What you might notice is a chatbot that answers a little faster when many people are using it at once. That is the whole benefit, and it depends on whether a company running a service adopts the method.
The numbers in the filing are small, tidy examples (112 KB versus 512 KB), so they show the idea works on paper but say little about a real data center. A 78% cut in swapped data sounds large, yet the filing does not say how much faster a full answer arrives.
The claims are broad, covering any processor group that splits requests, swaps attention results and then splits the second stage. For readers, that makes this a routine plumbing improvement with a clear and modest payoff: quicker replies, with no new feature to try.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Nvidia patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
26 drawing sheets from US 2026/0350352 A1 · click any drawing to enlarge
Show the other 18 sheets
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in