Microsoft Patents a Shortcut for the Math That Slows AI Chatbots
Microsoft has filed a patent application on a math shortcut for AI chatbots: replace one slow calculation with a small cheat sheet and a simple curve. The step it targets eats over 30 percent of the time a model spends on attention.
What Microsoft's softmax shortcut actually does
Ever wondered why a chatbot sometimes takes a beat before it starts answering? Part of the answer is a piece of math buried deep inside the model. Microsoft's filing says one step, called softmax, takes up over 30 percent of the time the model spends on attention, the part that decides which words in your text matter most to each other.
The trouble is partly a data-format hassle. Today's graphics chips do this math in a slower, high-precision mode, so numbers get converted back and forth, a bit like translating every sentence into another language and back again. Microsoft's fix splits each number into a whole part and a fractional part. The whole part is checked against a small cheat sheet, and the fractional part is estimated with a simple curve.
According to the filing, the answer lands close enough to the exact math that the model's quality holds up, while the chip stays in its faster mode.
… determining an integer portion of the exponentiation operation using the softmax approximation framework using a lookup table (LUT) to determine a value for the integer portion of the exponentiation operation; …
Translation: It uses a precomputed table to quickly handle whole numbers in the heavy math equation.
How the whole-number lookup and curve combine
Inside a large language model, an attention mechanism (the part that weighs which words matter most to each other) produces raw attention scores. Those scores go through softmax, which runs an exponentiation (raising the number e to a power) and then a normalization (rescaling so all the weights for a word add up to 1).
The exponent is the slow part. Current GPUs (the graphics chips that run AI) handle it in a high-precision 32-bit mode, while the rest of the work runs in faster 16-bit half precision, so data keeps getting converted back and forth. The filing does the exponent in half precision instead:
- Subtract the largest value in each row of scores, which keeps the numbers small.
- Split each score into a whole-number part and a decimal part between 0 and 1.
- Find the whole-number part in a compact lookup table.
- Estimate the decimal part with a polynomial (a short formula, degree 3 or less in some versions, fit with a standard curve-fitting method called least squares).
- Multiply the two results, then divide each value by the total so they sum to 1.
The filing also describes a sparsity step. Very negative scores shrink to almost nothing after exponentiation, so scores outside a chosen range (the example is 0 to -5) are set to zero. That keeps the table small. It also works with multi-head attention, where each head feeds its own scores into the same approximation.
… determining a decimal portion of the exponentiation operation by performing a polynomial approximation to determine a value for the decimal portion of the exponentiation operation; …
Translation: It uses a fast shortcut formula to approximate the remaining fractions.
Why a 30 percent bottleneck is worth trimming
If it works as described, the payoff is quicker and cheaper AI responses. Softmax takes over 30 percent of attention time, and the filing says its approximation keeps the model's accuracy intact while skipping the slow format conversions. Faster attention means less waiting for you and less computing for whoever runs the service.
The filing places the technique inside a cloud platform that powers content generation and AI-enabled search for apps. It also says the idea fits common attention designs. It gives no measured speedup, so how much you would notice is unknown. The claims do not name a specific chip, which leaves the method open to different hardware setups.
Microsoft's 77th filing in the AI assistant and agent work we've tracked since May connects to earlier applications like one on rounding AI numbers and one on AI site navigation.
The problem here is big. Every reply from a language model pays the softmax cost, and the filing puts that cost at over 30 percent of the attention time. A step that takes up that much of the work, repeated across millions of requests, is a fair target for an engineer.
The fix is small and matches the size of the problem: a short table, a three-term curve, and a rule for ignoring scores that round to zero. Nothing here promises a model twice as fast, and the filing offers no speed numbers to check.
Still, a simple, well-aimed trick on a costly step is how AI serving gets cheaper in practice. If the accuracy claim holds up, this earns more attention than its modest wording suggests.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Microsoft patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
10 drawing sheets from US 2026/0350372 A1 · click any drawing to enlarge
Show the other 2 sheets
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in