Nvidia Files Patent for a Chip Built to Make AI Chatbots Answer Faster
Between every step of a chatbot's reply, today's chips often park results in shared memory and fetch them again. Nvidia's patent filing describes a chip that skips the parking lot and passes results hand to hand.
What Nvidia's direct-handoff chip actually does
Every time a chatbot writes a word, the chip behind it works through dozens of layers, and a model like Llama 70B has 80 or more. On today's graphics chips, the results of each step usually get parked in shared memory and fetched again for the next one. All that back and forth adds delay, which you feel as a pause before the reply.
Nvidia's filing describes a different chip layout. Many small number-crunching blocks do the multiplication, and a second kind of block, called an IMLP tile, collects their partial answers, combines them, and passes the result straight to whichever block needs it next.
The wiring between blocks is set up ahead of time to match the AI model, so nothing has to stop at a shared memory in between. The goal is replies that arrive in real time while using less power.
… directly transmitting the local reduction results to at least one of a compute tile or a second IMLP tile through the configurable network to complete the LLI operation.
Translation: It sends calculated data straight to another chip section to finish the task quickly.
How the chip tiles pass results to each other
The patent splits a chip into two kinds of tiles. Compute tiles do the heavy math. Each holds a grid of small units (64 in one example) with its own local memory (SRAM, fast storage built next to the math circuits), vector registers (tiny scratch pads), and multiply-accumulate units (circuits that multiply numbers and add them to a running total, the basic move in AI math).
IMLP tiles (the filing's term for inter-matrix layer processing tiles) collect partial answers from a group of compute tiles and finish the job: sums, maximums, averages, and steps like normalization (rescaling numbers to a standard range). Claim 1 covers this finishing step. In the example, 16 compute tiles feed each IMLP tile.
The configurable network ties it together as three separate paths:
- Gather: carries partial results from compute tiles to their IMLP tile.
- Reduction: passes combined results between IMLP tiles in a tree shape.
- Distribution: broadcasts the final answer to the next compute tiles.
Connections are preset for each stage of the model, so data goes register to register without a shared memory stop. In the example system, regular GPUs read your prompt and these chips write the reply, word by word.
A configurable network transmits data directly from source to destination register(s) within compute and IMLP tiles.
Translation: A custom routing system sends information straight where it needs to go without slowing down.
Why faster chatbot replies depend on this
The practical payoff is speed you can feel. Voice assistants, live translation and driving software all need answers in a blink, and the filing lists those uses by name. Every trip to shared memory adds delay, so cutting those trips could make replies quicker and cheaper to run.
The filing also shows how Nvidia frames the job. Its example pairs ordinary GPUs, which read your prompt, with separate chips built to write the reply. That split suggests the reply phase of a chatbot is getting its own hardware thinking, with everything from the tile layout to the wiring tuned for it.
Nvidia's 64th filing we've tracked in the AI chip wars since July adds to a run that includes running big models with little memory and sharing one chatbot across chips.
Claim 1 describes a four-step loop: a helper tile receives values over a shared network, holds them in a register, runs a calculation, and sends a condensed result directly to another tile. No specific chip, no named application, no particular model size. That stripped-down language is deliberate.
That breadth means the claim would reach any chip following the same pattern, whether it powers a voice assistant, a medical scanner, or a car's navigation system. The dependent claims add narrowing details like averaging or summing across values, but the core claim sits above all of them.
Claims 16 and 19 restate the same arrangement as a physical system and as software instructions, giving the same idea three separate legal hooks. If claim 1 survives review unchanged, it fences off a wide stretch of modern inference hardware that sits inside nearly every AI product shipping today.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Nvidia patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
22 drawing sheets from US 2026/0349953 A1 · click any drawing to enlarge
Show the other 14 sheets
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in