IBM Files Patent for a Grade on How Well You Asked Your AI
IBM has filed a patent application for a system that grades your question before an AI chatbot answers it. A badly phrased question gets flagged or rewritten instead of producing a confident wrong answer.
What IBM's question-scoring system actually does
You're typing a question into your company's AI assistant, the one hooked up to thousands of internal documents. The answer comes back vague, and you can't tell whether the AI is dim or you just asked it badly.
IBM's patent application goes after that problem. Before the AI answers, the system gives your question a score. Ahead of time, an AI reads the documents and writes sample questions about each section. Your question gets compared to the closest of those samples. If it lines up well enough, the AI goes ahead and answers.
If it doesn't, the system can rewrite your question, adding background pulled from the documents, and test it again. It repeats until the score is good or it hits a set number of tries.
… modifying, by the one or more processors, the prompt via one or more artificial intelligence (AI) agents of an agentic Retrieval Augmented Generation (RAG) analyzing multiple embeddings in the vector database in relation to the question to determine contextual content relating to the question and modifying the prompt based on the contextual content; …
Translation: AI agents tweak your question by checking past examples in a database to add helpful context.
How IBM scores a question against sample ones
The system starts by building a library. Documents are split into chunks (paragraphs, pages, sections or sentences), and a language model writes sample questions about each chunk. Each question is turned into an embedding, a long list of numbers that captures its meaning, and stored in a vector database (a search index organized by meaning instead of exact keywords).
When a question comes in, it is cleaned up into a prompt (typos fixed, formatting standardized) and converted to an embedding too. In the main claimed version, AI agents also analyze the stored embeddings and add context to the prompt. This is called agentic RAG: RAG (retrieval augmented generation) means pulling relevant background text into a question, and "agentic" means AI helpers decide what to pull.
Next, the system fetches the N stored sample questions closest to the prompt, where N is at least 2. The filing's example uses the top 3. It measures how close each one is, using a word-overlap measure such as a Rouge score (which counts matching word sequences), then averages the results. In one example the score runs from 0 to 1, with a pass mark around 0.70 to 0.80.
If the score clears the bar, the prompt goes to the language model for an answer. If not, the system can stop or loop: rewrite the prompt, re-score it, and repeat up to a maximum number of tries. A variance check, which asks whether the matches are consistently close, can be added too.
A closeness metric score equal to an average of the N closeness metrics is computed.
Translation: The system averages out how well your phrasing matches known good questions.
What a question score means for chatbot users
If you've ever gotten a mushy answer from a chatbot wired to company documents, the cause is often the question. People rarely know the terms their own documents use. A score that flags a weak question, or fixes it automatically, would catch that before the AI replies with something confident and wrong.
The patent examiner has issued a final rejection of this application, so nothing here is approved. IBM can still amend the claims or appeal. Even so, the filing shows how much effort goes into making document-based AI assistants reliable, not just fast.
IBM's 76th filing we've tracked since May in the AI guardrails race continues a pattern seen in their earlier work on highlighting which words swayed a decision and an AI that quizzes itself.
The benefit you would notice is a chatbot that tells you your question missed before it hands you a smooth, plausible answer built on the wrong paragraph. That failure is the annoying one, because you often can't tell the answer is off.
A score of 0.75 doesn't mean much on its own. It helps only if the system also steers you toward a better question, and the claims describe the AI rewriting the question for you, which could be the more useful half. The score also compares word overlap with sample questions another AI wrote, so a fair question phrased in unusual words could score poorly.
Still, between a vague answer and a clear "try asking it this way," most people would take the second. Whether this particular method survives the examiner is a separate matter.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time IBM patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
7 drawing sheets from US 2026/0350096 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in