Microsoft Patent Shrinks the Word Pool First to Cut Text Generation Lag

By Patentlyze Team · Updated Jul 24, 2026

Every time an AI model generates a response, it considers a massive list of possible words for every single position in that response. Microsoft's new patent proposes doing a quick pre-filter pass so the model only has to look at a much smaller, more relevant slice of that list.

How Microsoft's vocabulary-filtering trick speeds up AI answers

Imagine a game-show contestant who has to pick the right answer from a list of a million options. It would take forever. But if a producer first narrows that list down to a hundred relevant choices, the contestant can answer almost instantly. That's the basic idea behind this Microsoft patent.

When an AI generates text, it normally scans through a huge dictionary of possible words (or word fragments) at every step. This patent describes a system that first reads your full question, uses it to assemble a much shorter list of relevant words, and then builds the answer using only that shorter list. Less scanning means the model reaches its answer faster and uses less computing power to get there.

Microsoft also describes letting the model generate the entire response in a single pass rather than word by word, which can speed things up further. The patent even supports tokens that represent whole multi-word phrases, not just single words, giving the model bigger building blocks to work with.

How the model encodes queries and filters candidate tokens

The patent describes a two-stage encoding process. When a query arrives, the model encodes it into two kinds of internal representations:

• A vocabulary-guiding embedding (a numeric fingerprint of the whole question, used to decide which words are even in play)
• A sequence of response-part embeddings (one embedding per chunk of the answer that needs to be filled in)

The vocabulary-guiding embedding is then mapped onto a reduced-size vocabulary, a shortlist drawn from a larger target vocabulary of tokens. Tokens here are the units the model thinks in: they can be word fragments, full words, or even multi-word phrases. Crucially, the shortlist is specific to the incoming query, so a question about weather pulls different candidate tokens than a question about finance.

Next, each response-part embedding is matched against that shortlist to produce a ranked group of candidate tokens for that position. The model picks from those ranked candidates to construct that piece of the answer.

The patent supports two generation modes: non-autoregressive (all positions filled in one simultaneous pass, fast but less flexible) and autoregressive (positions filled one at a time, each informed by previous choices). Both modes benefit from the pre-filtering step because the token search space is already much smaller.

What faster, cheaper AI inference means for Microsoft's products

AI inference, meaning the moment a model actually answers your question, is one of the biggest ongoing costs for any company running large language models at scale. Anything that reduces the number of comparisons the model makes per response directly cuts those costs and speeds up the time before an answer appears on your screen. For Microsoft, which embeds AI generation in Copilot, Bing, Azure OpenAI Service, and a growing set of enterprise tools, even modest per-query savings multiply into significant real-world impact.

The multi-word token angle is also interesting: if the model can treat a common phrase like "quarterly earnings" as a single unit, it can skip several individual word-selection steps at once. That's a form of compression that gets more valuable as response lengths grow, which matters a lot for business applications that generate long documents or structured reports.

Editorial commentary on a publicly published patent application. Not legal advice.