Red Hat Wins a Patent for Trimming Old Chat History Before an AI Answers You
Every time you message a chatbot, it rereads your entire conversation from the top. Red Hat has now been granted a patent on a way to cut that pile down to the parts that still matter.
What Red Hat's chat-trimming patent does
Imagine you've spent an hour with an AI assistant, planning a trip, polishing a job application, and settling a dinner-party argument. Every time you send a new message, the assistant rereads the whole conversation from the start. The longer you talk, the more it has to reread.
Red Hat has now been granted a patent on a fix. Before the AI answers, a separate program compares your newest question with everything said earlier. Pieces that have nothing to do with your question, or that simply repeat something, get tossed.
What's left is a shorter version of the chat that goes to the AI. The filing says this can shrink the input by 10%, 15%, 25% or more, which should mean faster answers and fewer made-up ones.
… generating a truncated embedding based on the second vector embedding, wherein generating the truncated embedding involves executing a filtering process that excludes a portion of the second vector embedding from the truncated embedding based on a comparison between the portion of the second vector embedding and at least a portion of the first vector embedding …
Translation: The system trims down old chat history by comparing past messages to your current question and cutting out the irrelevant parts.
How old messages get compared and dropped
The patent starts from a quirk of chatbots: the underlying model, known as a large language model (LLM), has no memory of its own. It only seems to remember because the app sends the whole conversation back with every new question. Long chats hit the model's size limit, slow it down, and cost more on services that charge by length.
Red Hat's approach converts both your newest question and the stored chat history into vector embeddings (lists of numbers that capture what a piece of text means). A separate embedding model, not the chatbot itself, does this conversion.
A filtering process then compares pieces of the history, such as a phrase, sentence or paragraph, against the question using a similarity score (the filing names cosine distance and Levenshtein distance):
- If a piece scores as too far from the question, it is dropped as irrelevant.
- If it scores as nearly identical, it is dropped as a duplicate.
- Everything else goes into a truncated embedding, the slimmed-down history.
The slimmed history is either fed to the model directly in number form or converted back to text using a saved mapping, the embedding model in reverse, or a second helper model. The surviving history is also saved as the starting point for the next round, so the chat keeps getting trimmed as it grows.
Reducing the size of the input prompt can decrease latency, execution time, and hallucinations associated with the LLM.
Translation: Trimming the chat history makes the AI faster and less likely to make things up.
Why shorter chat memory means cheaper AI
If you use chatbots for long sessions, the practical payoff is faster replies and fewer answers that drift or contradict something you said ten topics ago. For companies that pay per input token (the small chunks of text a model reads), a shorter prompt also means a smaller bill. The filing also points out that the usual alternative, chopping off the oldest messages, can throw away context you still need.
This is already a granted patent, though the final granted claims may read a little differently from the published application described here. Because the method runs on a server between the chat app and the model, it could in principle sit in front of almost any chatbot service.
This is the sixth IBM filing we've tracked since July in our assistants that remember you watchlist, building on one that rewrites your prompts and one that predicts your clicks.
Of everything in this filing, the most useful detail is how little it needs to work. It runs as software only, sits between the chat window and the model, and leaves the model itself untouched.
The shortest road to a product is a team dropping this into an existing chatbot service as a quiet efficiency step. The building blocks are a similarity score, two adjustable cutoff numbers, and a lookup table connecting those numbers back to the original text.
The real work is tuning those cutoffs, because set them wrong and the assistant either forgets something important or saves almost nothing. The filing points to savings of 10% to 25%, which makes this a behind-the-scenes improvement rather than a headline feature.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time IBM patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
4 drawing sheets from US 2026/0339929 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in