Nvidia Files Patent to Regroup AI Chips in the Middle of Each Chatbot Reply
Nvidia's patent filing describes taking one group of AI chips and reorganizing it on the fly, so the chips use one team layout for looking back over a chat and a different one for deciding the next word.
What Nvidia's chip-regrouping patent actually does
A chatbot is halfway through answering you, and before it picks each next word it has to look back over the whole conversation. The longer that conversation gets, the more notes it has to keep, and those notes can overwhelm a single chip's memory.
Nvidia's patent filing describes a way to split that job across a group of chips, then reshuffle the group partway through each step. First, the chips divide the notes between them so no one chip is buried. Then they regroup into a different team layout built for the next stage, where the model works out what comes next.
You would never see any of this. If it works as described, the same hardware could handle longer conversations without chips sitting idle or running out of memory.
reconfiguring a pool of processing units between a first parallelism mode for attention computation and a second parallelism mode for feedforward computation in a language model inference system …
Translation: Switching AI hardware setups halfway through a chatbot response to handle different types of math.
How the GPUs swap between two team layouts
The filing covers a pool of processing units (in practice, GPUs, the chips that run AI) that switches between two arrangements while a language model writes its answer one token (a word or word fragment) at a time.
In the first arrangement, used for attention (the step where the model looks back over everything said so far), the model's running notes, called the KV cache, are split by position in the conversation. Each chip holds a slice and works on it alone. The chips then share partial results plus two bookkeeping numbers, a maximum score and a normalizing total, so a correction step can produce the same answer a single chip would have.
In the second arrangement, used for the feedforward step (where each token passes through the model's neural networks), the same chips regroup along two dimensions: tensor parallelism (several chips share the math of one network) and expert parallelism (different specialist sub-networks sit on different chip groups). The total chip count equals those two numbers multiplied. Each token is sent to its top one or two experts.
Models with no experts simply set the expert dimension to one and fall back to ordinary tensor parallelism. The filing says a scheduler can switch modes per step or per layer based on latency goals, and can overlap batches of requests, one in each mode.
… reconfigured between different parallelism modes during inference. In a first mode for attention computation, contextual memory data is distributed across processing units. After attention computation, the units are reprovisioned into a second mode …
Translation: The system shifts the chips into a new formation right after the AI finishes reading the conversation history.
What this means for running long AI chats
If you use a chatbot, the cost of remembering a long conversation lands on the hardware. The filing says that pile of notes grows with every word and, for models aimed at millions of tokens, can limit performance. Spreading it across all the chips, then regrouping for the next stage, is meant to keep every chip busy.
The filing also says the same layout switch should work for ordinary models and expert-based models through a single code path, and that it could apply to video and multimodal models, which keep even bigger memories. For Nvidia, which sells both the chips and much of the software around them, that flexibility matters to data centers running many different kinds of models.
Nvidia's 60th filing we've tracked since July in the AI chip wars watchlist, this follows one on sharing partial AI results and one on cutting chip sorting time.
Of everything in this filing, the path to a product is short. Nothing here calls for a new chip. The filing describes a scheduler, a set of settings, and a pattern of data swaps between GPUs that already exist.
That points to software as the fastest route. A program that serves chatbots could adopt the layout switch in an update, and the filing says one code path can handle both ordinary and expert-based models.
What has to exist first is the plumbing: chips connected fast enough that the regrouping does not eat the savings, and a model large enough to benefit. The filing gives no measured speedup, so how much anyone would notice is unknown. If it ships, you would most likely feel it as quicker replies, with no new button to press.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Nvidia patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
14 drawing sheets from US 2026/0350380 A1 · click any drawing to enlarge
Show the other 6 sheets
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in