IBM Files Patent for an AI Referee That Picks the Best Chatbot Answer
Ask a chatbot about this week's weather and it may tell you it can't know. IBM's patent application describes a referee that collects answers from several systems, chatbots and search tools among them, and picks the one a human expert would pick.
What IBM's answer-picking AI actually does
A chatbot is asked which company has the quantum computer with the most qubits. It names a machine from 2021 and admits things may have changed. A plain web search, asked the same question, returns the newer record holder. Neither tool is wrong everywhere, and you rarely know which one to trust for a given question.
IBM's patent application describes a referee that sits in front of both. Your question goes to several systems, such as chatbots and search tools, and each writes an answer. A set of neural networks (software loosely modeled on the brain) then picks the answer a human expert would have picked, using your question plus details like your profile and the time you asked.
The expert picks do a second job: they correct the chatbot itself. When an expert fixes an answer, the fix counts as a penalty for the chatbot, and over many rounds it gets retrained to answer better.
an access component that accesses input data comprising a prompt and data associated with the prompt; and a response selection component that selects, based on the input data, a response to the prompt from a collection of responses by employing deep imitation learning.
Translation: The system uses deep imitation learning to look at a user prompt and pick the best answer from multiple chatbot options.
How IBM trains the judge on expert choices
The claims center on a response selector. It takes your prompt plus related data (the filing mentions a user profile, time stamps and GPS data) and picks one answer from a pool. The pool holds answers written by different models, which the filing calls knowledge-based systems: large language models (LLMs, the engines behind chatbots) and information retrieval systems (search engines and similar tools). The pool can hold answers from any number of such models.
The selector is trained with deep imitation learning, meaning a neural network learns by watching demonstrations. Subject matter experts (SMEs), who can be people or other software, rate or correct earlier answers and show which one they would have chosen. Those decision sequences become training data. The filing says imitation helps because writing a scoring rule for "best answer" across very different models is hard.
The dependent claims add a feedback loop:
- Experts are asked for ratings or corrections, stored in a structured format.
- The feedback is combined, checked for common errors, and turned into training pairs, such as a prompt plus its corrected answer.
- Good feedback becomes positive reward signals and bad feedback becomes penalties, which retrain the LLM through reinforcement learning (learning by trial and error guided by rewards).
- The retrained model is tested on held-back data or against the old version (A/B testing), and the cycle starts again.
Why a chatbot referee could matter to you
If this worked as described, you would not have to guess whether a chatbot or a search engine is better for a given question. A model trained on expert picks would make that call for you, and it could take into account details like the time you asked or your profile. The filing points to gaps it wants to cover: chatbots working from old training data, and chatbots saying "I can't know" on questions a search tool handles easily.
For companies, the bigger piece is the loop. Expert corrections flow back into the chatbot so it improves over time, and the filing says the cycle can restart automatically when new weak spots show up. The filing frames this as business value for an organization, though it names no product and gives no accuracy numbers.
That makes this IBM's 21st filing we've tracked since May in our AI models working in teams watch, which already includes one on models competing to answer questions and one on AI models debating security alerts.
Asking several systems the same question means running several systems for every prompt, which adds waiting time and computing cost before the referee even starts. The filing gives no numbers on either.
The referee also learns from experts, so it inherits their habits and blind spots. If the experts are rushed or disagree with each other, the referee copies the mess. Expert time is the expensive ingredient, because someone has to keep rating and correcting answers for the loop to keep turning.
For a company with experts on staff that cares about a right answer more than a fast one, the trade looks worthwhile. For a casual chatbot, the extra cost is hard to justify. The filing sketches a workflow and shows no test results.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time IBM patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
10 drawing sheets from US 2026/0340406 A1 · click any drawing to enlarge
Show the other 2 sheets
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in