Nvidia Files Patent for Hardware That Makes AI Chatbots Reply Faster
Nvidia's filing describes a chip that never parks a chatbot's in-progress work in shared memory. Each result goes straight to the next place that needs it, over wiring laid out before the model starts running.
What Nvidia's chatbot-speed chip actually does
Every time a chatbot writes a word, the computer behind it runs your question through a long stack of steps, 80 or more for a big model like Llama 70B. Today's graphics chips do each step fast, but they keep shuffling results to and from a shared memory between steps. Those trips add up, and you feel them as a pause before the reply.
Nvidia's filing describes a chip that skips most of those trips. The work is cut into many small blocks, each with its own little memory holding the model's numbers. Nearby helper blocks add up the pieces and pass the answer straight to whichever block needs it next, over wiring laid out before the model starts running.
In the setup described, ordinary graphics chips read your prompt, and boards of these new chips write the answer.
distributing input data across a number of compute tiles organized into groups of compute tiles to compute tensor operations for at least a processing operation of a neural network model using the input data …
Translation: Splitting the input across many processing tiles to run AI math in parallel.
How the tiles pass results with no memory stop
The filing targets the decode phase of a chatbot, when the model writes its answer one small piece of text (a token) at a time. The system it describes pairs graphics boards, which handle prefill (reading your prompt), with boards of new low-latency inference (LLI) chips for decode. Each board holds a 4x6 array of 24 chips.
The smallest building block is a basic unit: a slice of fast local memory (SRAM), a few vector registers (tiny scratch slots for numbers), and a multiply-accumulate unit (a circuit that multiplies numbers and keeps a running total). Many basic units form a compute tile, which keeps the model's weights and its attention memory (the key and value vectors) close to the math.
Three separate wiring networks do the moving:
- Gather: sends each tile's partial answer to a helper tile.
- Reduction: combines those helper results, for example finding a global maximum or sum, in a tree shape.
- Distribution: broadcasts the finished result directly into registers of the tiles that need it next.
The helper is called an IMLP tile (inter-matrix layer processing). How many tiles share a job depends on the shape of the math, meaning the size of the grid of numbers. The wiring is set up ahead of time for the model and changes stage to stage. In one example, the chip has about 2,000 compute tiles, and pipelining hides all the waiting time of the softmax step (the step that turns attention scores into weights).
A low-latency processing system reduces data movement, improves memory bandwidth utilization, and enables, e.g., scalable, low-latency neural network inference.
Translation: Cutting down on data travel speeds up AI chatbot response times.
Why Nvidia is splitting chatbot work across chips
If you have waited on a chatbot reply or a live translation, some of that pause comes from hardware shuffling numbers around, not from the math itself. This design attacks the shuffling. The filing also claims energy savings, since long trips to big shared memories burn power. It lists voice assistants, real-time translation, robotics and autonomous driving among the uses.
For Nvidia, the filing shows how it might split chatbot work: graphics chips for reading your prompt, a separate chip type for writing the answer, linked by its NVLink connections and scaled up across boards and racks. It is a patent application, and nothing in it says a product is coming. It does show where the company is looking for more speed once the usual graphics chip hits its limits.
Nvidia's 56th filing we've tracked in the AI chip wars since July follows patents like one to speed up chatbot replies and one to compress language models.
The price of this design is flexibility. Wiring set up ahead of time for one chatbot's exact shape is very fast. It also assumes you know that shape in advance, and the filing's answer to a new model is to reconfigure the connections and spread the work across more chips.
There is a hardware bill too. Keeping the model's numbers in tiny on-chip memories skips slow trips to shared memory, so a big model has to be sliced across many chips. The filing puts 24 on a board and links boards into racks. Add the regular graphics chips still needed to read your prompt, and an operator has two kinds of machines to buy and run.
For a company selling the speed of live answers, the trade pays off. A chatbot that replies with no pause is something customers feel right away, and a data center serving many people can absorb extra equipment far more easily than a person can absorb a delay. The open risk is whether fixed wiring ages well as chatbot designs keep changing.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Nvidia patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
22 drawing sheets from US 2026/0330205 A1 · click any drawing to enlarge
Show the other 14 sheets
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in