Nvidia Files Patent for Running Big AI Models on Graphics Cards With Little Memory
Your graphics card could start loading the next piece of an AI model while it is still working on the current one. Nvidia's filing uses that overlap so a model bigger than the card's memory can still answer without long waits.
What Nvidia's patent does for oversized AI models
Every time an AI helper on your PC answers a question, its whole brain has to sit in your graphics card's memory. If that brain is too big, the card runs one part, stops, fetches the next part, and waits again. Those pauses add up fast.
Nvidia's filing chops the AI model into chunks and sets aside two memory slots. While the card works through one chunk, the next chunk is already being copied into the other slot. When the first chunk finishes, the second is ready to go, and the slot that just freed up starts loading chunk number three.
The filing's own example is a game assistant that answers questions like "How do I tame it?" while sharing the card with the game. It stays inside a fixed memory budget so the game's frame rate doesn't suffer.
cause a second portion of a neural network to be loaded into memory concurrently with inferencing performed using a first portion of the neural network …
Translation: It swaps in the next piece of the AI model while the current piece is still working.
How Nvidia's two memory buffers swap AI chunks
The claims cover one core idea: load a second portion (a shard) of a neural network into memory at the same time the processor is running inference (getting an answer out of the trained model) on the first portion. Then use the first portion's output to run the second. The filing calls this pipelined sharding.
A software library splits the model along its layers. It reserves a fixed slice of the graphics card's memory (VRAM) for the key-value cache, a running scratchpad the model uses to remember earlier words in the conversation. The rest is divided into two equal buffers, and each shard is sized to fit in half of what's left. When a shard finishes, its output (the activations, meaning intermediate results) is copied into the other buffer. The third shard then loads into the space the first one vacated.
How much work the graphics card (GPU) does depends on the link between it and the main processor (CPU):
- Fast link (the filing cites NVLink-C2C at 900 GB/s): every shard runs on the GPU, for both reading the prompt and writing the answer.
- Slower link (PCIe): pipelining handles the prompt-reading phase. For the answer-writing phase, the scratchpad is copied to the CPU, and roughly 35 to 40 percent of the shards run there while the GPU pipelines the rest.
The library checks the link type by asking the graphics driver. In one example, a model gets a 1 GB budget split into shards of 500 MB. The filing also says the model needn't be pruned, distilled or quantized (shrunk) to fit.
… cause the loaded second portion to be used to perform inference using output of the first portion of the neural network …
Translation: The newly loaded section then takes the results from the previous step and keeps going.
Why this matters for game assistants on your PC
The practical payoff is for AI features that have to share a graphics card with something else, such as a game. The filing's example keeps the assistant inside a fixed memory budget so the game still gets the rest. It also says the full-size model can run as-is, without the accuracy loss that usually comes with shrinking it.
There are limits, and the filing admits them. If copying takes longer than the work it hides, even the partial version may gain nothing, and the system falls back to a fixed split between CPU and GPU. How much you would gain depends on how quickly your machine moves data between its parts. The filing measures success by how fast the first word appears and how many words arrive per second.
Nvidia's 63rd filing we've tracked since July in our AI chip wars watch adds to earlier work on splitting a chatbot's workload and spreading its long memory, exploring more ways chips coordinate.
The person who benefits most is anyone who wants an AI helper running on their own PC while a game is on screen. The failure it targets is easy to picture: the assistant hogs graphics memory, the game stutters, or the answer takes forever to start. The filing tracks exactly those two numbers, how fast the first word appears and how many words follow each second.
You would likely never see the technique itself. You would see a helper that responds in a reasonable time without having been shrunk to fit, which the filing says it avoids needing.
The filing is candid about a limit. On a slower connection between processor and graphics card, copying can eat the gain, and the processor has to take roughly a third of the work. Whether you would notice an improvement depends on how fast your PC moves data around.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Nvidia patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
31 drawing sheets from US 2026/0350343 A1 · click any drawing to enlarge
Show the other 23 sheets
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in