IBM Patents a Test That Picks the Best AI for Each Chatbot Job
Companies have plenty of AI models to choose from for a chatbot, and judging which one fits a given job is harder than it sounds. IBM's patent application turns that choice into a scorecard.

- Publication number
- US 2026/0368473 A1(US20260368473A1)
- Applicant
- International Business Machines Corporation
- Filed · Published
- May 19, 2025 · Nov 19, 2026
- Inventors
- Puspendu Chatterjee, Ilan Sezhiyan Jayaraman, Bidhu Ranjan Sahoo
- Status when we published
- Waiting for an examiner
- Document
- 20 claims
- US classification
- 706/11
What IBM's chatbot model scorecard actually does
A bank's help chatbot could run on any of five different AI models, and nobody on the team is sure which one answers customers best. Picking by gut feeling is risky, and IBM says the usual ways of comparing models don't give consistent results from one task to the next.
IBM's patent application describes a scorecard. You set aside some test questions that have known good answers and let every candidate model answer them. Then the system checks how close each reply lands to the good answer. "Close" is measured several ways: same meaning, same words, same sentence structure, and same fit with the conversation. Each check gets a weight, such as 40% for matching content.
The model with the best middle score across all the test questions wins the job. The system can then recommend it to you, or simply switch to it.
… evaluating, based at least in part on the processing of the at least one set of validation data, performance of the multiple artificial intelligence models using a weighted combination of two or more similarity measures in conjunction with one or more performance metrics …
Translation: The system scores how well each AI model handles test data using a mix of different comparison tools and accuracy standards.
How four similarity checks get blended into one score
The method starts with validation data (test questions with known good answers) tied to one conversation task, such as answering customer questions, summarizing text, or judging sentiment. Several candidate models, either large language models (the engines behind modern chatbots) or older natural language processing models, each produce answers.
Each answer is compared with the reference answer using up to four similarity checks:
- Semantic: embeddings (numbers that capture meaning) plus cosine similarity (a closeness score between two sets of numbers).
- Content: text split into word pieces plus Jaccard similarity (how much the two word sets overlap).
- Syntactic: word order and sentence structure.
- Contextual: context-aware embeddings plus cosine similarity.
A weighting scheme blends these into one score. The filing's example is 40% content, 25% contextual, 20% semantic and 15% syntactic, tuned to the conversation system in question. Standard performance metrics such as accuracy, BLEU and ROUGE (scores for translation and summary quality) can be used alongside, as can real-world ones like user satisfaction, speed and cost savings.
For each model, the system takes the median (the middle value) of its scores across all validation inputs, and the highest median wins. In the filing's example, about 80 of 100 data points train the models and about 20 are held back for testing. Automated follow-ups include recommending the winner, running the task with it, or retuning models through retraining or fine-tuning.
… selecting, based at least in part on the evaluating of the performance of the multiple artificial intelligence models, one of the multiple artificial intelligence models to be used in executing at least one instance of the at least one designated automated conversation system task …
Translation: The best model for a specific chatbot job is chosen automatically based on those performance scores.
What this means for companies running chatbots
If a company runs a help chatbot, the AI model behind it decides whether you get a useful answer or a confusing one. This filing describes a way to test candidate models on the company's own sample questions before one goes live, and to choose by score. It can also weigh practical factors such as response time, memory use and user satisfaction, so a faster model could beat a slightly more polished one.
For any business juggling several models, the appeal is plain: swapping models could be checked automatically instead of argued over. The filing says the weights and metrics can be customized per task, and new models can be added to the comparison. You would likely never see this system. You would only notice it if a chatbot got better at understanding you.
This is the 24th IBM filing we've tracked since May on AI models working in teams, following one on grading a small AI's code and one on faster answers over encrypted data.
Source: US 2026/0368473 A1 at the USPTO (PDF). Full patent text and figures are in the official publication.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time IBM patent news breaks.
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings · 4 sheets
4 drawing sheets from US 2026/0368473 A1 · click any drawing to enlarge

No comments yet