Microsoft Patent Reveals How AI Systems Can Answer Faster by Reusing Predictions
Every time an AI chatbot types out a response, a lot of behind-the-scenes checking slows things down. Microsoft's new patent describes a way to do less of that checking without sacrificing accuracy.
How Microsoft's expanded token matching speeds up AI replies
A large language model generates text one word (or word-piece) at a time, and each step takes compute power. On a busy server, that adds up to noticeable lag between your question and the answer.
Microsoft's patent tackles this with a two-model trick. A small, fast "draft" model races ahead and suggests a batch of words. A larger, more accurate "verifying" model then checks those suggestions. Normally, any word the verifier rejects gets thrown out entirely. This patent adds a second-chance rule: even if a draft word doesn't pass the strict check, it can still be accepted if it meets a looser backup criterion.
The result is that more draft words survive the review process, so the big model has to do less original generation work. That means your response arrives faster and the server uses less energy doing it.
… verifying correctness of the draft tokens in the set of draft tokens by comparing probability information generated by the first machine-trained model with probability information generated by the second machine-trained model …
Translation: The main model checks the helper model's work by comparing their mathematical predictions.
How the verifier decides which draft tokens to keep or rescue
The technique builds on an existing approach called speculative decoding (where a small "draft" model proposes tokens and a large "verifier" model approves or rejects them in batch). The novelty here is an extra acceptance layer.
- Draft phase: The small model rapidly produces a set of candidate tokens (the atomic units of text an AI works with, roughly syllable-sized chunks of words).
- Verification phase: The large model runs its own probability estimates over each draft token and compares them to the draft model's estimates. A token passes if the distributions are close enough under the standard rejection-sampling rule.
- Expanded matching phase: Any token that fails the standard check gets a second look under a specified matching criterion (a configurable, looser rule). If it passes that, it is still accepted rather than discarded.
- Iteration: Whichever tokens survive are fed back to the draft model to seed the next batch, and the cycle repeats until the full response is complete.
The patent is model-agnostic: it doesn't require any particular architecture. The "matching criterion" is described as configurable, meaning teams could tune how permissive the second-chance rule is depending on their accuracy-vs-speed trade-off.
The technique has the overall effect of expanding a number of draft tokens that are accepted, thereby reducing use of resources and deceasing latency.
Translation: This method accepts more predicted words to speed up responses and save computing power.
What faster AI inference means for everyday chatbot users
For anyone using a Microsoft AI product (Copilot in Word, GitHub Copilot, Azure OpenAI-backed services), the payoff is a shorter pause before text appears. That gap between submitting a prompt and seeing words stream back is directly tied to how efficiently the underlying model can run. Fewer rejected draft tokens means fewer expensive "from scratch" generation steps by the large model.
Microsoft's run of inference-efficiency filings points to a broader effort to make large models cheaper to serve at scale. On the cost side, accepting more draft tokens per cycle reduces how often the big, energy-hungry verifier has to do its own generation, which matters when you're running millions of queries per day.
This is the 19th Microsoft filing we've tracked since May on AI models working in teams, following applications on two models teaming up for chat and giving AI teammates a visual presence.
The concrete payoff here is real but subtle. If you use a Microsoft AI assistant today, you already experience speculative decoding working in the background. This patent would, if implemented, shave additional milliseconds off the wait by rescuing words that would otherwise have been thrown away and regenerated at higher cost.
For most users, that difference will land below the threshold of conscious notice. Where it does show up is in long-document drafting or coding tasks where the model generates hundreds of tokens in sequence and the time savings compound.
The approach is incremental rather than transformative. It doesn't change how models are trained or how accurate they are. It's a refinement to the plumbing that happens between models, and that kind of work tends to matter most to the engineers paying cloud compute bills rather than the people typing prompts.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
10 drawing sheets from US 2026/0260065 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →