IBM
IBM · AI / ML · Filed May 7, 2025 · USPTO published Nov 12, 2026 · This week's release
IBM Patents a Panel of AI Judges to Grade Chatbot Answers
IBM's patent builds a courtroom for chatbot answers: a panel of AI judges scores each reply, and a higher court of super judges breaks any tie.
- Publication number
- US 2026/0359578 A1(US20260359578A1)
- Applicant
- International Business Machines Corporation
- Filed · Published
- May 7, 2025 · Nov 12, 2026
- Inventors
- Bo WEN
- Status when we published
- Waiting for an examiner
- Document
- 20 claims
- US classification
- 704/9
- Examiner
- SERRAGUARD, SEAN ERIN (Art Unit 2657)
What IBM's panel of AI judges actually does
Every time you ask a chatbot a question, you're trusting that it didn't make something up. For a dinner recipe, that's a small risk. For medical advice or computer code, a confident wrong answer can do real damage.
IBM's patent describes a grading system for those answers. A panel of AI judges, each set up to look at something different such as accuracy, safety or clear writing, scores the response. If the judges mostly agree, that's the verdict.
If they split, the answer goes up to a second tier of super judges that settle the dispute. Their rulings then train the first panel, so the judges get better at their job over time.
… measuring, by the generative system, a consensus among the evaluations of the ensemble of judge agents using a differentiator; resolving, by the generative system, one or more low consensus cases among the evaluations using a second layer of super judge agents; …
Translation: The system checks if the AI judges agree and calls in a panel of superior judges when they disagree.
How the judges' disagreement triggers a second tier
The system starts after a chatbot has produced an answer. That answer goes to a first layer of judge agents (separate AI programs, each told to grade the answer from one angle). The filing's examples include clinical accuracy, safety, fit with medical guidelines and clear writing. The judges can run on different large language models (the AI engines behind chatbots), from different vendors or with different settings, so they don't all think alike. The filing suggests an odd number of judges, such as 3, 5 or 7, to avoid ties.
A component called the differentiator then checks how much the judges agree. One method averages the scores and calculates the variance (a measure of how spread out they are), then compares it to a set threshold. Another compares how similar the judges' written reasoning is in meaning.
High agreement ends the process. Low agreement calls in the super judges, which compare the judges' evaluations in pairs and can consult a domain knowledge base, such as medical guidelines. A weighted vote, where each judge counts according to its record on test data, produces the final evaluation.
Finally, the system tunes the judges using reinforcement learning (a trial-and-reward training method) based on the super judges' rulings. The filing says this can run continuously.
Why a chatbot referee matters for health and code
Chatbots that confidently invent facts become a bigger problem the more people rely on them for things like medical questions or code. The filing's pitch is an automated referee that can grade answers at a scale human experts can't match, since expert review is slow and expensive. If it works as described, a checking layer like this could catch bad answers before they reach you, or show where a chatbot keeps going wrong.
The patent doesn't name a product. It frames the system as an evaluation tool for companies running chatbots, and the claims reach any system that generates content. For you, the effect would show up indirectly, as chatbots in sensitive fields that get tested more often and more carefully.
IBM's 78th filing we've tracked since May in our AI safety guardrails work adds to earlier applications on keeping AI on facts and grading how you ask AI, pushing further into AI oversight.
Source: US 2026/0359578 A1 at the USPTO (PDF). Full patent text and figures are in the official publication.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time IBM patent news breaks.
Add Patentlyze on Google
One click.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings · 7 sheets
7 drawing sheets from US 2026/0359578 A1 · click any drawing to enlarge
More from IBM
All 470 IBM patents →
More in AI assistant & agent
All 413 AI assistant & agent patents →
OpenAI
OpenAI Patents a Rulebook Its AI Reads Only When a Touchy Topic Comes Up
Amazon
Amazon Files Patent for a Fact-Check on the Links AI Chatbots Cite
Microsoft
Microsoft Patents a System to Keep Its Chatbot Behaving Consistently Over Time
Google
No comments yet