Get the newsletter
AMD · AI / ML · Filed Dec 23, 2024 · USPTO published Jun 25, 2026 · Jun 25 release

AMD Patent Reduces Hardware Storage Costs by Generating AI Memory Data On Demand

Running a large AI model eats memory fast, and one of the biggest culprits is a cache of calculations the model drags along with every word it generates. AMD's new patent proposes throwing that cache away and recomputing it on the fly instead.

By · Updated · 2 min read

AMD Patent: Shrinking AI Memory by Recomputing Key Data, figure from US 2026/0178832 A1
Figure from the official USPTO publication.
Publication number
US 2026/0178832 A1(US20260178832A1)
Applicant
ADVANCED MICRO DEVICES, INC.
Filed · Published
Dec 23, 2024 · Jun 25, 2026
Inventors
Shaizeen Aga, Akila Subramaniam, Suchita Pati
Status when we published
Waiting for an examiner
Document
20 claims
US classification
704/9
Examiner
CHAVEZ, RODRIGO A (Art Unit 2658)
AI assistant & agent

How AMD decides what to discard and recompute each token

Generative AI models like large language models process text in layers, and each layer relies on two sets of vectors called keys and values (the KV in KV cache). These are intermediate calculations that summarize the context the model has seen so far. Normally, every key and value computed during a generation pass is kept in memory so later layers can refer back to them.

AMD's patent describes a different approach: selective recomputation. Instead of holding onto every KV vector after it is first used, the system discards them. When the next token (the next word or word-piece) needs to be generated, the system recomputes only the specific KV vectors that the current layer requires, at the moment it needs them.

The key design decision is the word selective. Not every vector gets thrown away; the system decides which ones are worth recalculating versus which ones are cheap enough to store. That judgment call is what separates this from simply running the full model twice.

  • Generate a token using KV vectors from all relevant model layers.
  • Discard those vectors rather than holding them in the KV cache.
  • On the next token, recompute only the vectors needed, layer by layer, just before each layer consumes them.

The net effect is a significant reduction in peak memory footprint, trading some additional compute cycles for far less memory pressure.

What this means for running large AI models on tighter hardware

Memory is the main bottleneck for running large AI models on real hardware. The KV cache is often the single largest consumer of GPU or accelerator memory during inference, and it scales with both model size and the length of the conversation. A patent that reduces that footprint without changing the underlying model weights is directly useful for deploying AI at scale or on smaller devices.

For AMD, this fits into the company's push to compete with Nvidia on AI inference workloads. If AMD accelerators can run the same models while using less memory, that is a practical advantage for data center customers weighing chip options. It also opens the door to running longer context windows on hardware that would otherwise run out of room.

Source: US 2026/0178832 A1 at the USPTO (PDF). Full patent text and figures are in the official publication.

Get our take in your Top Stories

Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time AMD patent news breaks.

Add Patentlyze on Google One click.
There are more where this came from

We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.

More from AMD

All 119 AMD patents →

More in AI assistant & agent

All 413 AI assistant & agent patents →