Microsoft Files Patent to Trim Long Text Before an AI Chatbot Reads It
Microsoft's filing describes a filter that scores every sentence in a long document and hands a chatbot only the ones that matter. The goal is a faster, cheaper answer without losing the point.
What Microsoft's text-trimming filter actually does
Today, every AI chatbot has a cap on how much text you can feed it at once, and the more you hand over, the more computing power and waiting time the answer costs. Some apps cope by keeping only the most recent chunk and dropping the rest, which can toss out the one detail you needed.
Microsoft's patent filing takes a different route. Before the chatbot sees anything, a separate model reads your long document or email thread and gives each sentence a relevance score. Only the highest-scoring sentences move on, kept in their original order with their wording untouched.
The filing reports that one test case finished in 4.0 seconds instead of 7.6, and that human reviewers mostly preferred the summaries produced this way.
… selecting a subset of the sentences based on the scores to produce compressed information having fewer sentences than the input information as originally received by the receiving …
Translation: It picks the most important sentences and drops the rest to make the text shorter.
How overlapping windows score each sentence
The system works as a filter placed in front of a generative model (an AI that writes new text, like a chatbot). It takes a long input and splits it into overlapping windows, which are fixed-length chunks where each one starts partway through the one before it. Each window goes to its own encoder (a piece of a neural network that turns words into numbers), so the chunks are processed in parallel instead of one after another.
Every sentence gets a special marker at its start. After the first stage, each marker carries a numeric summary of its sentence. A second stage, the sentence-level encoder, looks at all those summaries together and gives each sentence a score for how useful it is to understanding the whole text.
Then the selection step runs:
- Rank the sentences by score.
- Keep the top ones until they fit the chatbot's input limit (the filing's example is 2,000 units of text).
- Put the kept sentences back in their original order.
- Optionally always keep the first sentence, since it often sets the topic.
Both stages are trained together on documents paired with the sentences best suited to a summary. The chatbot itself is trained separately, so the filter can sit in front of different models. The chatbot can also rewrite the kept sentences into a shorter summary.
… enables the machine-trained generative model to process the original input information with reduced resources and latency …
Translation: This helps the AI run faster and use less computing power by reading a shorter version of the text.
Why shorter prompts could mean faster AI answers
If you have ever pasted a long email thread into an AI assistant and waited, this is the problem the filing goes after. The filing's own numbers show memory use per chip climbing from 2.6 GB at 2,000 units of text to 28.2 GB at 8,000, and the 8,000-unit example took 7.6 seconds without the filter versus 4.0 with it. The filter itself took about 0.4 seconds in one case.
The filing says the filter works with question answering, chat and summarizing, including questions like "what date does Project ABC launch, according to this email thread?" For you, the payoff would be answers that arrive sooner. For Microsoft, it would be a cheaper bill every time someone asks an AI about a long document.
That makes this Microsoft's 73rd filing we've tracked in AI assistant and agent patents since May, adding to work like button labels for AI and an AI-built answer key.
Long inputs are expensive for AI. The filing's own figures show memory per chip jumping from 2.6 GB to 28.2 GB as the text grows, and every extra second of delay is a second you spend watching a blinking cursor. That cost shows up every time anyone asks a chatbot about a long document, so it adds up fast.
Cutting the input before the model sees it is a sensible match for that cost, because the biggest savings come from giving the model less to read. The risk is that a filter can misjudge. If it scores a key sentence low, the chatbot never learns that sentence existed. The reported tests, where reviewers mostly preferred the results, are encouraging, though they are Microsoft's own.
Verdict: a modest fix for a large, everyday expense. You would experience it as an assistant that answers a bit sooner.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Microsoft patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
10 drawing sheets from US 2026/0340017 A1 · click any drawing to enlarge
Show the other 2 sheets
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in