Nvidia Patents a Way to Run Two AI Networks at Once to Speed Up Text Generation
Every AI chatbot you've ever used generates its response one word at a time, waiting until it's done before moving on to the next step. Nvidia's latest patent tries to break that bottleneck by running two separate neural networks at the same time.
How Nvidia cuts the wait time on AI-generated text
Every time you send a message to an AI assistant and watch the dots spin, the model is working through two jobs back to back: first it processes your question, then it writes out the answer one word at a time. Those jobs normally queue up in a line, each one waiting for the one before it to finish.
Nvidia's patent describes a setup where a second AI network starts preparing for later work while the first network is still writing out the answer. Instead of waiting, the system hands off each word to the second network as soon as it's ready, so both networks are busy at the same time.
The practical goal is a faster response, especially in situations where AI is running locally on a chip rather than on a distant server. For on-device AI, every millisecond saved matters because the hardware is more limited than a data center.
cause one or more first neural networks to begin generating a set of tokens; and cause one or more second neural networks to perform one or more prefill operations on a first token of the set of tokens in parallel with the generation of a subsequent token of the set of tokens.
Translation: One AI starts making words while a second AI gets ready for the next steps at the same time.
How two neural networks split the token workload
The patent describes a two-network pipeline designed to overlap two phases of AI text generation that normally run one after the other.
The first phase is called prefill: the model reads your input and builds an internal summary of what you asked. The second phase is token generation: the model writes the answer word by word (technically, unit by unit, where each unit is called a token). In a standard setup, prefill has to finish completely before generation starts, and nothing else can happen in the meantime.
Nvidia's approach splits this across two separate networks:
- A first neural network begins generating the output tokens (the words of the answer) right away.
- A second neural network runs prefill on each token as soon as it's produced, in parallel with the first network generating the next token.
The key word in the patent's claim is parallel. By overlapping these two operations, the chip is doing useful work on both fronts simultaneously rather than leaving one unit idle while the other finishes. This is essentially a pipeline architecture (the same idea used in processor design for decades) applied to the specific sequence of operations inside a large language model.
What faster AI output means for real products
For everyday users, this is about response speed. AI assistants that feel slow or choppy are harder to trust and less useful in real-time tasks like live translation, voice assistants, or code completion. Shaving time off the generation pipeline, even by a small fraction, compounds across every query.
Nvidia has been filing around on-device and inference-efficiency AI since 2023 For Nvidia's hardware business, an efficient inference pipeline is a direct selling point: chips that generate tokens faster for the same power budget are easier to sell to both cloud providers and device makers. This patent sits squarely in that argument.
Nvidia's 42nd filing we've tracked since July in our AI chip wars watchlist builds on one reserving resources ahead of time and one routing tasks to workers.
Running two networks at once means both have to fit in memory at the same time, and on a phone or laptop that headroom simply may not exist. Power draw rises, memory gets squeezed, and the speed gains the design promises can evaporate before they reach the devices where fast responses matter most to ordinary people.
There is also a hidden timing problem: the second network needs to be ready the moment the first one hands off a result, and if either side runs slow, they end up waiting on each other, clawing back much of the time the whole approach was designed to save.
The underlying idea, borrowed from manufacturing where the next step starts before the previous one finishes, pays off handsomely when resources are plentiful. On a data center server, this trade reads as worth it. On the constrained hardware most people actually use, the math is far less favorable, and that gap is the real question this design leaves open.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
10 drawing sheets from US 2026/0278386 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →