Microsoft Files Patent for Rounding AI Numbers More Carefully to Speed Up Chatbots
A chatbot's few oversized numbers can wreck the shortcuts that make it fast. Microsoft's patent gives every row and column its own reference point so the shortcut survives.
What Microsoft's number-squeezing patent does
Ever shrunk a giant photo so it would email faster, only to find it came out blurry? AI chatbots face a version of that problem. The numbers inside them are stored in a very precise format that eats memory and computing power, and squeezing them into a smaller format saves effort but can wreck the answers.
Microsoft's patent tackles one touchy spot: the part of a chatbot that decides which words in your question matter most to each other. Some numbers there are often far bigger than the rest, and those outliers throw off the usual squeezing methods. The filing gives each row or column its own reference point, so the big ones don't drown out the small ones.
The payoff, according to the filing, is that the chatbot can do its math on the squeezed numbers directly, without converting them back first. That skips a slow step and cuts memory use.
… quantizing the query data matrix and the key data matrix using a quantization framework of the large language model to convert floating point values of the query data matrix and the key data matrix to integer values to generate a quantized query data matrix and a quantized key data matrix …
Translation: Microsoft's system converts complex decimal numbers into simpler whole numbers to make the AI run faster.
How each row gets its own reference point
The patent covers a way to run the attention step of a large language model (LLM, the type of AI behind chatbots) using small whole numbers instead of long decimals. Attention is where the model scores how relevant each word is to the others, using two grids of numbers called the query matrix and the key matrix.
Shrinking 32-bit decimals into 8-bit integers is called quantization. It needs a zero point, a reference value that anchors the shrunken range. Usual methods use one zero point for the whole grid. Here, each channel (one feature of the data, stored as a row or column) gets its own, picked as the smallest value in that row or column. In the description's example, a single shared zero point is -202.1, while the per-row version uses -16.9, -202.1 and 19.
The claimed steps run like this:
- Turn the text into query and key grids.
- Convert both to integers, with one zero point per channel.
- Compute attention scores directly on the integers, with no converting back.
- Run a softmax (a step that turns scores into percentages) to get each key's attention weight.
The description says the extra terms add no computing steps and only a negligible bump in memory. It also says the approach lets GPUs use their faster tensor cores (chip units built for grid math) for the exponent math inside softmax.
… analyzing the quantized matrices using the attention mechanism to determine attention scores without dequantizing data of the quantized matrices …
Translation: The AI processes these simplified numbers directly without converting them back first.
Why cheaper chatbot math matters for Microsoft
The practical stake is cost. Chatbots are expensive to run, and the filing itself names computing, memory, power and even water use as the burdens of operating AI models. If the attention step gets faster and lighter, the same model could answer quicker or fit on hardware with less memory and computing power. That is a plain win for anyone who pays for, or waits on, AI replies.
The description ties the technique to a cloud platform that serves apps for design, communication and collaboration, all of which call on language models. For you, the result would show up as speed and price, since nothing about how you use the apps would change. The filing describes it as a speed-up for models in general, with no named product.
Microsoft's 23rd filing we've tracked since July in the AI chip wars watchlist, it follows one on removing a traffic bottleneck and one on cheaper math for AI parts.
Every time you squeeze numbers into a smaller format, you give up some precision. The filing admits it: its own worked example shows the converted-back numbers don't exactly match the originals. Microsoft's bet is that a little rounding error is a fair price for skipping the conversion step.
The cost of this design is bookkeeping. Each row or column needs its own reference value stored and folded into the math, which the description says adds negligible memory and no extra steps. It offers no speed or accuracy test results, so that claim rests on the company's word.
On paper the trade reads as worth it. Small accuracy losses inside a chatbot's math are usually tolerable, and shaving cost off every reply adds up at cloud scale. The weak spot would be a model whose outliers are so extreme that even per-row reference points can't tame them.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Microsoft patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
14 drawing sheets from US 2026/0350374 A1 · click any drawing to enlarge
Show the other 6 sheets
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in