Nvidia Files Patent for Routing Chatbot Questions to the Server That Already Did the Reading
Every time you send a chatbot a follow-up, it may reread the same text from scratch. Nvidia's patent application aims to send your question to a machine that already has the notes.

- Publication number
- US 2026/0368481 A1(US20260368481A1)
- Applicant
- NVIDIA Corporation
- Filed · Published
- May 19, 2025 · Nov 19, 2026
- Inventors
- James Allan Douglas Cameron, Ryan Michael Olson, Bartley D. Richardson, Michael D. Demoret
- Status when we published
- Waiting for an examiner
- Document
- 20 claims
- US classification
- 706/17
What Nvidia's chatbot-memory patent actually does
Imagine you work at a company whose AI assistant starts every question with the same 20 pages of instructions and documents. Each time, the AI reads all 20 pages from scratch, as if it had never seen them.
Nvidia's patent application tries to stop that waste. When an AI answers a question, it does a lot of up-front reading, and normally that work gets thrown away. Here, the system keeps it. When a new question arrives, it checks which computer in a big server farm has already done the reading for the same opening text, and sends your question there.
In theory, that means less repeated work, so answers should come back sooner and use less computing power. The system also checks that the chosen computer still has room to take the job, so a machine with the right notes doesn't get buried.
one or more circuits to cause one or more large language models (LLMs) to perform inferencing for a prompt using one or more previously-used tokens identified by the one or more LLMs to be re-used based on a hash-based representation of the prompt and a hash-based representation of the one or more tokens.
Translation: Hardware that speeds up chatbot answers by reusing previously processed text pieces matched through digital fingerprints.
How Nvidia matches a prompt to the right server
When a large language model (LLM, the engine behind a chatbot) handles a prompt, it chops the text into tokens (small pieces such as words or word fragments). It then computes intermediate results for each token and stores them in a KV cache (key-value cache, a short-term memory the model reads back as it writes an answer). Normally that work is discarded. This filing keeps it for later requests.
To find a match quickly, the system turns the prompt's tokens into a hash chain: a fingerprint for each piece, where every fingerprint depends on the ones before it. Each server's saved tokens are summarized the same way. Comparing the chains shows how much of the prompt a server has already processed. The filing's example: for a prompt A→B→C→D→E, device 1 holds A→B→C→D, device 2 holds A→B, and device 3 holds A, so device 1 gets the job.
The selector can weigh more than overlap:
- Reuse score: how long the matching run is.
- Capacity: whether the machine has cache space for the new tokens the answer will create.
- Stacking: many near-identical requests can share one machine, because only new tokens need storage.
The claims cover a processor, a method and a system. Dependent claims add reuse during the prefill phase (the long reading step before the model starts writing) and requests from agent applications.
… one or more tokens may be previously identified by one or more LLMs and the one or more tokens previously identified by the one or more LLMs may be caused to be reused by the one or more LLMs to perform inferencing.
Translation: The system recycles earlier computational work so the AI doesn't have to read the same material twice.
Why Nvidia cares about repeated chatbot work
For you, the payoff would be indirect: chatbots and AI assistants that wait less before they start answering, and services that cost less to run. Many AI products send the same long instructions or documents with every request, so skipping that repeated reading adds up fast across millions of requests.
For Nvidia, the filing sits at the layer where its chips and data center software meet. It covers how requests get routed across many machines, not a new chip. Keep in mind this is a patent application, not a product announcement, and the filing doesn't name any specific product.
That makes this Nvidia's 18th filing we've tracked since July in our AI models working in teams watchlist, after their patent on a photo-describing team of AI models and one on fixing floating objects in 3D scenes.
Source: US 2026/0368481 A1 at the USPTO (PDF). Full patent text and figures are in the official publication.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Nvidia patent news breaks.
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings · 26 sheets
26 drawing sheets from US 2026/0368481 A1 · click any drawing to enlarge
Show the other 18 sheets

No comments yet