Microsoft
Microsoft · Chip · Filed Jul 9, 2026 · USPTO published Nov 12, 2026 · This week's release
Microsoft Patents Memory Chips That Only Activate When an AI Generates a Response
When a chatbot reads your question and then writes its answer, it is doing two different jobs. This Microsoft filing gives each job a different kind of math.
- Publication number
- US 2026/0359563 A1(US20260359563A1)
- Applicant
- Microsoft Technology Licensing, LLC
- Filed · Published
- Jul 9, 2026 · Nov 12, 2026
- Inventors
- Ori LASLO, Gilad KIRSHENBOIM
- Status when we published
- Waiting for an examiner
- Document
- 20 claims
- US classification
- 704/9
- Parent application
- Continuation of 18/483,307 (filed Oct 9, 2023)(18483307)
What Microsoft's switchable memory math does
Ever notice how a chatbot takes in your whole question at once, then types its answer one small piece at a time? Those are two very different jobs for the computer underneath, and this Microsoft patent application treats them that way.
Reading your question is heavy math done on lots of words together. Writing the answer is lighter math, but the computer keeps fetching the same huge set of learned numbers from memory for every new word, so the memory becomes the traffic jam. Some memory chips can do a little math themselves to cut that traffic, though their math is clumsy compared with the main processor's.
Microsoft's filing switches that built-in math off while your question is being read and on while the answer is being written. The processor's strong math handles the first job, and the memory's faster delivery helps with the second.
… based on the prompt stage operation, causing the prompt stage operation to bypass an in-memory compute engine of the memory; executing the prompt stage operation without using the in-memory compute engine; …
Translation: The system skips the special memory hardware while reading the user prompt.
How the bypass engine picks a mode for each step
Generative AI (software that writes text, like a chatbot) answers you in two stages. In the prompt stage, it takes in your whole input at once as a batch of word pieces called tokens. In the auto-regressive stage, it writes the reply one token at a time, each based on the ones before.
The memory chip (DRAM, the fast short-term memory next to the processor) holds the model's learned numbers, called weights. The filing describes memory with a small in-memory compute engine that does simple math inside the chip, mainly a dot product (multiply two lists of numbers and add up the results). That cuts the data traveling to the processor. The catch is that memory chips are built for storage, so their math is slow and power hungry next to the processor's.
A bypass engine decides, operation by operation, which mode to use:
- Prompt stage: the processor reads data out of memory, skips the in-memory math, and does the calculations itself.
- Auto-regressive stage: the in-memory engine computes the dot product and sends back the result, so less data crosses to the processor.
- Switching: a dedicated signal, a control register, a pre-defined address map, or two types of read commands can flip the mode.
The filing also says some auto-regressive steps that need no dot product can skip the in-memory engine too.
A compute management engine supports dynamically switching between two modes of operation for an inference phase of a generative artificial AI model.
Translation: A manager switches how the memory works during different steps of AI generation.
What this means for running chatbots at scale
Behind every chatbot reply is hardware doing a lot of repeated fetching, since the model's weights get pulled from memory again for every new token. The filing says in-memory math has been held back because it is inefficient, and that turning it off during the prompt stage removes that drawback. It also claims lower processor workload and smaller memory needs than earlier systems.
The description places this inside a cloud AI system serving apps such as chat interfaces, so the likely beneficiaries are data centers running large language models. It is a continuation of an application filed in October 2023, so the idea has been in the works for a while. As with any patent application, nothing here says it will show up in a product.
Microsoft's 27th filing we've tracked in AI chip wars since July adds to a run that includes a light-based AI chip and training on idle gaming PCs.
Source: US 2026/0359563 A1 at the USPTO (PDF). Full patent text and figures are in the official publication.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Microsoft patent news breaks.
Add Patentlyze on Google
One click.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings · 9 sheets
9 drawing sheets from US 2026/0359563 A1 · click any drawing to enlarge
Show the other 1 sheets
More from Microsoft
All 646 Microsoft patents →
More in Chip
All 645 Chip patents →
Samsung
Samsung Files Patent for Memory Chips That Do Some of an AI's Math Themselves
Qualcomm
Qualcomm Patents a Way to Let Memory Chips Handle Multiple Jobs at Once
OpenAI
OpenAI Patents a Way to Search AI Memory Without Moving All the Data
Samsung
No comments yet