AMD · AI / ML · Filed May 8, 2025 · USPTO published Nov 12, 2026 · This week's release

AMD Files Patent for Chips That Adjust How Far Ahead a Chatbot Guesses

Chatbots speed up their answers by guessing a few words ahead and having a bigger model check the guesses, but how far ahead to guess is usually fixed. AMD's patent application has the chip adjust that distance on the fly, and steer around lengths where checking suddenly gets slow.

By Patentlyze Team · Updated Nov 12, 2026 · 3 min read

An AI chip that speeds up responses by automatically changing how many words a chatbot guesses in advance. Drawing from patent filing US 2026/0359674 A1. — An AI chip that speeds up responses by automatically changing how many words a chatbot guesses in advance.All 6 drawings ↓

Publication number: US 2026/0359674 A1(US20260359674A1)
Applicant: Advanced Micro Devices, Inc.
Filed · Published: May 8, 2025 · Nov 12, 2026
Inventors: Sonali SINGH, Karthik Ramu SANGAIAH
Status when we published: Waiting for an examiner
Document: 20 claims
US classification: 706/46

AI assistant & agent

What AMD's adaptive guess-ahead patent actually does

A junior writer drafts the next five words of a sentence, and a senior editor checks all five at once, crossing out any that are wrong. That is roughly how many AI chatbots speed up their replies, using a trick called speculative decoding. The number of words the junior guesses ahead is usually set in advance and left alone.

AMD's patent application would make that number change while you wait. If the editor keeps approving the guesses, the system lets the junior guess one more word ahead. If the editor keeps rejecting them, it cuts back by one.

There is a second wrinkle. On a given chip, checking a longer draft can suddenly take much more time at certain lengths, like a toll that jumps at particular mile markers. The filing has the chip steer clear of those lengths, so your answer keeps arriving quickly.

From the filing · CLAIM 1

determine a moving average acceptance rate for draft length tokens sequentially generated for a large language model (LLM) inference call using speculative decoding (SpD) …

Translation: It tracks how often the AI accepts the words it guesses ahead of time.

How the chip avoids slow draft lengths

The patent covers circuitry inside a GPU or similar AI chip. In speculative decoding, a small draft model produces K tokens (word fragments) one after another, then the large target model checks all K in parallel. The filing says K is often between 2 and 10 tokens. The new part is that K changes during use.

The first piece is the draft length adjustment circuitry. It tracks a moving average acceptance rate (the share of drafted tokens the big model accepted, averaged over recent drafts) and compares it to a threshold. Above the threshold, K goes up; below it, K goes down. The main example moves K by one at a time.

The second piece is stepping point detection and adjustment circuitry. Stepping points are draft lengths where checking time jumps suddenly. The filing says they are measured ahead of time by profiling the model on the specific hardware and stored in a lookup table, not measured live. Then:

If raising K lands on a stepping point, K goes back to its previous value.
If lowering K lands on one, K drops one more step.
A hysteresis factor (a damper that stops K from flipping back and forth) keeps a backlog count, so K only jumps higher after high acceptance persists.

The claims add that the damper can sit at the midpoint of two consecutive stepping points, or be set from a running average of acceptance rates.

From the filing · THE ABSTRACT

The draft length is incremented when the moving average acceptance rate is greater than the threshold acceptance rate. The draft length is decremented when the moving average acceptance rate is less than the threshold acceptance rate.

Translation: The chip lets the AI guess more words when it is accurate and fewer when it makes mistakes.

What this could mean for AI chatbot speed

If you use a chatbot or a coding assistant, the wait before text appears is the cost you feel most. The filing names real-time AI applications like these as its target, and its aim is to keep the speedup from speculative decoding from eroding when the system guesses too far ahead or lands on a slow spot for the chip.

For AMD, the interesting angle is that the fix lives in the accelerator circuitry and is tuned to the hardware. The filing says slow spots differ from chip to chip, so a setup that works well on one GPU could run poorly on another. A chip that knows its own quirks could help AMD make the case for its hardware in AI serving. Nothing in the filing ties this to a specific product.

AMD's 49th filing we've tracked since May in the AI chip wars watchlist, following work on faster custom model switching and image AI memory cuts, adds another piece to the picture.

Editorial take

The problem here is a leak. Speculative decoding promises a big speedup on paper, and the filing admits real-world results come in lower, partly because checking longer drafts costs more time than expected. For a chatbot that answers millions of questions a day, small leaks add up to real money and real waiting.

The approach fits that size. Nudging the guess length by one word at a time and steering around slow spots that were measured in advance is simple bookkeeping, cheap enough to build into the hardware. Because those slow spots are measured once for each model on each type of chip, the setup has to be redone whenever the model or the chip changes.

The filing gives no numbers for how much faster replies get, so treat it as a sensible tune-up with a modest payoff.

Source: US 2026/0359674 A1 at the USPTO (PDF). Full patent text and figures are in the official publication.

Get our take in your Top Stories

Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time AMD patent news breaks.

Add Patentlyze on Google One click.

There are more where this came from

We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.

The drawings · 6 sheets

6 drawing sheets from US 2026/0359674 A1 · click any drawing to enlarge

Chip parts that tune how far an AI guesses, drawing sheet 1 from US 2026/0359674 A1 — Sheet 1 Chip parts that tune how far an AI guesses

Two ways to adjust AI guess length, drawing sheet 2 from US 2026/0359674 A1 — Sheet 2 Two ways to adjust AI guess length

Core chip layout that handles heavy AI calculations, drawing sheet 3 from US 2026/0359674 A1 — Sheet 3 Core chip layout that handles heavy AI calculations

How an AI changes its guess count based on accuracy, drawing sheet 4 from US 2026/0359674 A1 — Sheet 4 How an AI changes its guess count based on accuracy

Fine-tuning guess lengths so the chip avoids sudden slowdowns, drawing sheet 5 from US 2026/0359674 A1 — Sheet 5 Fine-tuning guess lengths so the chip avoids sudden slowdowns

Tracking recent guess success rates to keep responses fast, drawing sheet 6 from US 2026/0359674 A1 — Sheet 6 Tracking recent guess success rates to keep responses fast

Watchlists this filing belongs to

All watchlists →

The AI chip warsCross-company · 411 filings

Reader comments

The drawings · 6 sheets

More from AMD

AMD Patents a Technique to Cut Memory Traffic During AI Chip Processing

AMD Patent Stores AI Model Data Efficiently to Speed Up Processing

AMD Patent Reduces Hardware Storage Costs by Generating AI Memory Data On Demand

AMD Patents Software That Reads Your Messages and Automatically Handles Business Tasks

More in AI assistant & agent

A New Nvidia Patent Reworks How Many Chips Share One Chatbot's Workload

A New Nvidia Patent Spreads a Chatbot's Long Memory Across Many Chips

Nvidia Files Patent to Regroup AI Chips in the Middle of Each Chatbot Reply

Microsoft Files Patent for Rounding AI Numbers More Carefully to Speed Up Chatbots

Watchlists this filing belongs to

No comments yet

Start the discussion Cancel reply