Get the newsletter
IBM · AI / ML · Filed May 19, 2025 · USPTO published Nov 19, 2026 · This week's release

IBM Patents a Test That Picks the Best AI for Each Chatbot Job

Companies have plenty of AI models to choose from for a chatbot, and judging which one fits a given job is harder than it sounds. IBM's patent application turns that choice into a scorecard.

A step-by-step test that trains multiple AI programs and scores their responses against sample data to find the best match for a specific job. Drawing from patent filing US 2026/0368473 A1.
A step-by-step test that trains multiple AI programs and scores their responses against sample data to find the best match for a specific job.All 4 drawings ↓
Publication number
US 2026/0368473 A1(US20260368473A1)
Applicant
International Business Machines Corporation
Filed · Published
May 19, 2025 · Nov 19, 2026
Inventors
Puspendu Chatterjee, Ilan Sezhiyan Jayaraman, Bidhu Ranjan Sahoo
Status when we published
Waiting for an examiner
Document
20 claims
US classification
706/11

What IBM's chatbot model scorecard actually does

A bank's help chatbot could run on any of five different AI models, and nobody on the team is sure which one answers customers best. Picking by gut feeling is risky, and IBM says the usual ways of comparing models don't give consistent results from one task to the next.

IBM's patent application describes a scorecard. You set aside some test questions that have known good answers and let every candidate model answer them. Then the system checks how close each reply lands to the good answer. "Close" is measured several ways: same meaning, same words, same sentence structure, and same fit with the conversation. Each check gets a weight, such as 40% for matching content.

The model with the best middle score across all the test questions wins the job. The system can then recommend it to you, or simply switch to it.

From the filing · CLAIM 1
… evaluating, based at least in part on the processing of the at least one set of validation data, performance of the multiple artificial intelligence models using a weighted combination of two or more similarity measures in conjunction with one or more performance metrics …

Translation: The system scores how well each AI model handles test data using a mix of different comparison tools and accuracy standards.

How four similarity checks get blended into one score

The method starts with validation data (test questions with known good answers) tied to one conversation task, such as answering customer questions, summarizing text, or judging sentiment. Several candidate models, either large language models (the engines behind modern chatbots) or older natural language processing models, each produce answers.

Each answer is compared with the reference answer using up to four similarity checks:

  • Semantic: embeddings (numbers that capture meaning) plus cosine similarity (a closeness score between two sets of numbers).
  • Content: text split into word pieces plus Jaccard similarity (how much the two word sets overlap).
  • Syntactic: word order and sentence structure.
  • Contextual: context-aware embeddings plus cosine similarity.

A weighting scheme blends these into one score. The filing's example is 40% content, 25% contextual, 20% semantic and 15% syntactic, tuned to the conversation system in question. Standard performance metrics such as accuracy, BLEU and ROUGE (scores for translation and summary quality) can be used alongside, as can real-world ones like user satisfaction, speed and cost savings.

For each model, the system takes the median (the middle value) of its scores across all validation inputs, and the highest median wins. In the filing's example, about 80 of 100 data points train the models and about 20 are held back for testing. Automated follow-ups include recommending the winner, running the task with it, or retuning models through retraining or fine-tuning.

From the filing · THE ABSTRACT
… selecting, based at least in part on the evaluating of the performance of the multiple artificial intelligence models, one of the multiple artificial intelligence models to be used in executing at least one instance of the at least one designated automated conversation system task …

Translation: The best model for a specific chatbot job is chosen automatically based on those performance scores.

What this means for companies running chatbots

If a company runs a help chatbot, the AI model behind it decides whether you get a useful answer or a confusing one. This filing describes a way to test candidate models on the company's own sample questions before one goes live, and to choose by score. It can also weigh practical factors such as response time, memory use and user satisfaction, so a faster model could beat a slightly more polished one.

For any business juggling several models, the appeal is plain: swapping models could be checked automatically instead of argued over. The filing says the weights and metrics can be customized per task, and new models can be added to the comparison. You would likely never see this system. You would only notice it if a chatbot got better at understanding you.

This is the 24th IBM filing we've tracked since May on AI models working in teams, following one on grading a small AI's code and one on faster answers over encrypted data.

Source: US 2026/0368473 A1 at the USPTO (PDF). Full patent text and figures are in the official publication.

Get our take in your Top Stories

Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time IBM patent news breaks.

There are more where this came from

We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.

The drawings · 4 sheets

4 drawing sheets from US 2026/0368473 A1 · click any drawing to enlarge

Testing several AI models to pick the best chatbot, drawing sheet 1 from US 2026/0368473 A1
Sheet 1 Testing several AI models to pick the best chatbot
Scoring model answers against test data to find matches, drawing sheet 2 from US 2026/0368473 A1
Sheet 2 Scoring model answers against test data to find matches
Steps for grading chatbot models on a specific job, drawing sheet 3 from US 2026/0368473 A1
Sheet 3 Steps for grading chatbot models on a specific job
Hardware and cloud networks running the model tests, drawing sheet 4 from US 2026/0368473 A1
Sheet 4 Hardware and cloud networks running the model tests
Patent filing page

More in AI assistant & agent

All 436 AI assistant & agent patents →

Watchlists this filing belongs to

All watchlists →
Reader comments

No comments yet

Start the discussion

Real name or a handle, either is fine. Comments are read by a person before they appear, so allow a little time. Keep it about the filing.