Get the newsletter
Nvidia · AI / ML · Filed May 19, 2025 · USPTO published Nov 19, 2026 · This week's release

Nvidia Files Patent for Routing Chatbot Questions to the Server That Already Did the Reading

Every time you send a chatbot a follow-up, it may reread the same text from scratch. Nvidia's patent application aims to send your question to a machine that already has the notes.

A central router sends incoming AI requests to the server that already has relevant pieces of text saved in its memory. Drawing from patent filing US 2026/0368481 A1.
A central router sends incoming AI requests to the server that already has relevant pieces of text saved in its memory.All 26 drawings ↓
Publication number
US 2026/0368481 A1(US20260368481A1)
Applicant
NVIDIA Corporation
Filed · Published
May 19, 2025 · Nov 19, 2026
Inventors
James Allan Douglas Cameron, Ryan Michael Olson, Bartley D. Richardson, Michael D. Demoret
Status when we published
Waiting for an examiner
Document
20 claims
US classification
706/17

What Nvidia's chatbot-memory patent actually does

Imagine you work at a company whose AI assistant starts every question with the same 20 pages of instructions and documents. Each time, the AI reads all 20 pages from scratch, as if it had never seen them.

Nvidia's patent application tries to stop that waste. When an AI answers a question, it does a lot of up-front reading, and normally that work gets thrown away. Here, the system keeps it. When a new question arrives, it checks which computer in a big server farm has already done the reading for the same opening text, and sends your question there.

In theory, that means less repeated work, so answers should come back sooner and use less computing power. The system also checks that the chosen computer still has room to take the job, so a machine with the right notes doesn't get buried.

From the filing · CLAIM 1
one or more circuits to cause one or more large language models (LLMs) to perform inferencing for a prompt using one or more previously-used tokens identified by the one or more LLMs to be re-used based on a hash-based representation of the prompt and a hash-based representation of the one or more tokens.

Translation: Hardware that speeds up chatbot answers by reusing previously processed text pieces matched through digital fingerprints.

How Nvidia matches a prompt to the right server

When a large language model (LLM, the engine behind a chatbot) handles a prompt, it chops the text into tokens (small pieces such as words or word fragments). It then computes intermediate results for each token and stores them in a KV cache (key-value cache, a short-term memory the model reads back as it writes an answer). Normally that work is discarded. This filing keeps it for later requests.

To find a match quickly, the system turns the prompt's tokens into a hash chain: a fingerprint for each piece, where every fingerprint depends on the ones before it. Each server's saved tokens are summarized the same way. Comparing the chains shows how much of the prompt a server has already processed. The filing's example: for a prompt A→B→C→D→E, device 1 holds A→B→C→D, device 2 holds A→B, and device 3 holds A, so device 1 gets the job.

The selector can weigh more than overlap:

  • Reuse score: how long the matching run is.
  • Capacity: whether the machine has cache space for the new tokens the answer will create.
  • Stacking: many near-identical requests can share one machine, because only new tokens need storage.

The claims cover a processor, a method and a system. Dependent claims add reuse during the prefill phase (the long reading step before the model starts writing) and requests from agent applications.

From the filing · THE ABSTRACT
… one or more tokens may be previously identified by one or more LLMs and the one or more tokens previously identified by the one or more LLMs may be caused to be reused by the one or more LLMs to perform inferencing.

Translation: The system recycles earlier computational work so the AI doesn't have to read the same material twice.

Why Nvidia cares about repeated chatbot work

For you, the payoff would be indirect: chatbots and AI assistants that wait less before they start answering, and services that cost less to run. Many AI products send the same long instructions or documents with every request, so skipping that repeated reading adds up fast across millions of requests.

For Nvidia, the filing sits at the layer where its chips and data center software meet. It covers how requests get routed across many machines, not a new chip. Keep in mind this is a patent application, not a product announcement, and the filing doesn't name any specific product.

That makes this Nvidia's 18th filing we've tracked since July in our AI models working in teams watchlist, after their patent on a photo-describing team of AI models and one on fixing floating objects in 3D scenes.

Source: US 2026/0368481 A1 at the USPTO (PDF). Full patent text and figures are in the official publication.

Get our take in your Top Stories

Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Nvidia patent news breaks.

There are more where this came from

We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.

The drawings · 26 sheets

26 drawing sheets from US 2026/0368481 A1 · click any drawing to enlarge

Routing your prompt to a server with related memory, drawing sheet 1 from US 2026/0368481 A1
Sheet 1 Routing your prompt to a server with related memory
Saving early reading work to speed up chatbot replies, drawing sheet 2 from US 2026/0368481 A1
Sheet 2 Saving early reading work to speed up chatbot replies
Connecting user chats with servers tracked for reusable text memory, drawing sheet 3 from US 2026/0368481 A1
Sheet 3 Connecting user chats with servers tracked for reusable text memory
Matching your prompt to memory to pick a server, drawing sheet 4 from US 2026/0368481 A1
Sheet 4 Matching your prompt to memory to pick a server
Steps for answering questions faster by reusing stored text data, drawing sheet 5 from US 2026/0368481 A1
Sheet 5 Steps for answering questions faster by reusing stored text data
Cloud data center hardware and management software running the AI, drawing sheet 6 from US 2026/0368481 A1
Sheet 6 Cloud data center hardware and management software running the AI
All in one processor chip combining compute cores and graphics, drawing sheet 7 from US 2026/0368481 A1
Sheet 7 All in one processor chip combining compute cores and graphics
Parallel graphics chip designed to speed up heavy AI computations, drawing sheet 8 from US 2026/0368481 A1
Sheet 8 Parallel graphics chip designed to speed up heavy AI computations
Show the other 18 sheets
Internal computing blocks inside a parallel graphics processor, drawing sheet 9 from US 2026/0368481 A1
Sheet 9 Internal computing blocks inside a parallel graphics processor
A hybrid chip combining standard cores and dedicated AI hardware, drawing sheet 10 from US 2026/0368481 A1
Sheet 10 A hybrid chip combining standard cores and dedicated AI hardware
Inside a multi-core central processor and its instruction units, drawing sheet 11 from US 2026/0368481 A1
Sheet 11 Inside a multi-core central processor and its instruction units
Dedicated AI accelerator chip layout with built-in memory, drawing sheet 12 from US 2026/0368481 A1
Sheet 12 Dedicated AI accelerator chip layout with built-in memory
Brain-inspired chip designed to process information like neurons, drawing sheet 13 from US 2026/0368481 A1
Sheet 13 Brain-inspired chip designed to process information like neurons
Network of connected server racks sharing computational memory, drawing sheet 14 from US 2026/0368481 A1
Sheet 14 Network of connected server racks sharing computational memory
Multi-die accelerator chip built for heavy parallel workloads, drawing sheet 15 from US 2026/0368481 A1
Sheet 15 Multi-die accelerator chip built for heavy parallel workloads
Central processor integrating security, graphics, and AI hardware, drawing sheet 16 from US 2026/0368481 A1
Sheet 16 Central processor integrating security, graphics, and AI hardware
Internal parts of a server processor core, drawing sheet 17 from US 2026/0368481 A1
Sheet 17 Internal parts of a server processor core
Specialized circuits for multiplying numbers in AI tasks, drawing sheet 18 from US 2026/0368481 A1
Sheet 18 Specialized circuits for multiplying numbers in AI tasks
An alternative processor core designed to run AI tasks, drawing sheet 19 from US 2026/0368481 A1
Sheet 19 An alternative processor core designed to run AI tasks
Grid layouts linking computing cores across a chip, drawing sheet 20 from US 2026/0368481 A1
Sheet 20 Grid layouts linking computing cores across a chip
Layers of software connecting an app to computer chips, drawing sheet 21 from US 2026/0368481 A1
Sheet 21 Layers of software connecting an app to computer chips
Splitting program code to run across different processors, drawing sheet 22 from US 2026/0368481 A1
Sheet 22 Splitting program code to run across different processors
Translating programs to run on different brands of chips, drawing sheet 23 from US 2026/0368481 A1
Sheet 23 Translating programs to run on different brands of chips
Sensors and computers powering a self-driving car, drawing sheet 24 from US 2026/0368481 A1
Sheet 24 Sensors and computers powering a self-driving car
Hardware setups used to train and run AI models, drawing sheet 25 from US 2026/0368481 A1
Sheet 25 Hardware setups used to train and run AI models
Teaching an AI model before putting it to work, drawing sheet 26 from US 2026/0368481 A1
Sheet 26 Teaching an AI model before putting it to work
Patent filing page

More in AI assistant & agent

All 436 AI assistant & agent patents →

Watchlists this filing belongs to

All watchlists →
Reader comments

No comments yet

Start the discussion

Real name or a handle, either is fine. Comments are read by a person before they appear, so allow a little time. Keep it about the filing.