Get the newsletter
Amazon · Chip · Filed Mar 31, 2025 · USPTO published Oct 1, 2026 · Oct 1 release

Amazon Patents Technology to Make AI Systems Run Faster on Its Own Hardware

Running a large AI model is expensive, partly because the model's internal settings take up enormous amounts of memory. Amazon has filed a patent for a chip-level system that compresses those settings on the fly, so the hardware never has to store or move the full data in the first place.

A machine learning accelerator connects to a host computer and cloud, featuring a compute engine with neural processing units and memory buffers. Drawing from patent filing US 2026/0300680 A1.
A machine learning accelerator connects to a host computer and cloud, featuring a compute engine with neural processing units and memory buffers.All 7 drawings ↓
Publication number
US 2026/0300680 A1(US20260300680A1)
Applicant
Amazon Technologies, Inc.
Filed · Published
Mar 31, 2025 · Oct 1, 2026
Inventors
Qiuwen Lou, Tong Qin, Sankalp Dayal, Tariq Afzal, Rahul Bakshi
Status when we published
Waiting for an examiner
Document
20 claims
US classification
706/27

How Amazon shrinks AI models without losing accuracy

An AI model sits at the center of almost every Amazon product today, from Alexa to product recommendations to cloud services. But behind those features are millions of numerical settings, called weights, that the model consults every time it thinks. Storing and reading all those numbers is slow and expensive.

Amazon's patent describes a system that keeps a compressed version of those weights in fast on-chip memory. When the AI chip needs a weight, it looks up a short code, decodes it into the full number right there on the chip, and uses it immediately. You get the accuracy of a full-size model without paying the memory bill for storing every number in its largest form.

The key is a small lookup table called a codebook. Instead of storing thousands of individual weight values, the chip stores short codes that point to entries in that table. A dedicated part of the chip decodes each code back into a usable number in real time, keeping the whole process fast.

From the filing · CLAIM 1
… programming, by the input selector, the ALU to decode the first code data using the first decoding algorithm; generating, by the ALU, a first full-precision weight vector by decoding the first code data, …

Translation: The hardware is configured to decode compressed neural network data back into useful values.

How the codebook lookup and ALU decoding actually work

The patent describes a neural network accelerator, a specialized chip designed to run AI models faster than a general-purpose processor can.

Inside that chip, the system works in four steps:

  • Index vector lookup: For each layer of the neural network, the chip reads a small set of short codes (stored as hexadecimal values, essentially compact numeric shorthand) from fast on-chip SRAM memory.
  • Codebook reference: Those short codes are used as pointers into a codebook, a pre-built table that maps each code to a richer block of data called code data.
  • Decoding: An input selector component picks the right decoding algorithm for that particular index vector, then programs the chip's arithmetic logic unit (ALU, the part that does math) to reconstruct a full-precision weight vector, meaning a set of weight values at full numerical detail.
  • Inference: A compute engine uses those reconstructed weights to calculate the layer's output, exactly as it would with uncompressed weights.

The claim is specific about the hardware: a dedicated input selector, a programmable ALU, and a compute engine all working together on a single accelerator chip. That hardware specificity is what distinguishes this from a software-only compression scheme.

From the filing · THE ABSTRACT
Devices and techniques are generally described for vectorwise weight quantization for neural networks. In some examples, a first index vector stored in volatile memory may be determined for a first layer of a neural network.

Translation: The system compresses neural network weights to make processing more efficient.

What this means for Amazon's custom AI chip ambitions

Memory bandwidth is one of the biggest bottlenecks in running large AI models. Every time a model processes something, it has to read millions of weight values from memory. Compressing those weights into codebook codes and decoding them on the chip means less data has to travel across the memory bus, which translates to faster responses and lower power draw on Amazon's own infrastructure.

Amazon builds its own AI chips under the Trainium and Inferentia product lines, and the pattern in Amazon's custom-chip filings points squarely at reducing dependence on third-party silicon for AI workloads. A patent like this, covering on-chip weight decoding at the hardware level, fits that effort directly.

Amazon's 11th filing we've tracked since May in the AI chip wars follows one on compressing AI math and one on surviving server crashes during training.

Source: US 2026/0300680 A1 at the USPTO (PDF). Full patent text and figures are in the official publication.

Get our take in your Top Stories

Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Amazon patent news breaks.

There are more where this came from

We read the new patents from Big Tech, carmakers and more and send you the ones worth knowing. Plain English, free, every week.

The drawings · 7 sheets

7 drawing sheets from US 2026/0300680 A1 · click any drawing to enlarge

Machine learning accelerator system overview, drawing sheet 1 from US 2026/0300680 A1
Sheet 1 Machine learning accelerator system overview
SRAM and codebook memory, drawing sheet 2 from US 2026/0300680 A1
Sheet 2 SRAM and codebook memory
Quantized weights decoding process, drawing sheet 3 from US 2026/0300680 A1
Sheet 3 Quantized weights decoding process
Neural network weight decoding steps, drawing sheet 4 from US 2026/0300680 A1
Sheet 4 Neural network weight decoding steps
Device with machine learning accelerator, drawing sheet 5 from US 2026/0300680 A1
Sheet 5 Device with machine learning accelerator
Decoding logic and ALU components, drawing sheet 6 from US 2026/0300680 A1
Sheet 6 Decoding logic and ALU components
Decoding logic and ALU with shift, drawing sheet 7 from US 2026/0300680 A1
Sheet 7 Decoding logic and ALU with shift
Patent filing page

Watchlists this filing belongs to

All watchlists →
Reader comments

No comments yet

Start the discussion

Real name or a handle, either is fine. Comments are read by a person before they appear, so allow a little time. Keep it about the filing.