IBM Files Patent to Find Which Examples Are Confusing Your AI Chatbot
Give a chatbot five sample answers and one of them may be sabotaging the other four. IBM's patent application describes a way to find out which one.
What IBM's example-testing patent actually does
Imagine you are teaching a chatbot to translate computer code from Python to Java. You paste in five sample translations to show it what you want, and the answers still come out wrong. Which of the five samples is the troublemaker? You have no way to tell, so you shuffle things around and hope.
IBM's patent application describes a tool that sits between you and the AI and does that detective work for you. It tries many different mixes of your samples, checks how well each mix performs, and gives every sample a score for how much it helped or hurt.
In the filing's own example, one sample adds six points, another adds four, and a third subtracts two. The tool then drops the dud and sends your request again with only the helpful samples.
… constructing a set of test prompts using the set of context examples, wherein each test prompt of the set of test prompts comprises a different combination of context examples; …
Translation: It builds multiple test questions mixing different examples to see how the AI reacts.
How IBM scores each example in a prompt
The patent covers a method that intercepts a prompt on its way to a large language model (the type of AI behind chatbots) and pulls out the context examples, which are the sample input and output pairs included to show the model what to do.
From there it runs a loop:
- Build a batch of test prompts, each holding a different combination of those examples.
- Run every test prompt through the model and record how well each result performs.
- Compute an attribution score for each example (a number for its marginal contribution, meaning how much the results change with that example in the mix versus out of it).
- Rewrite the original prompt to include only the subset of examples with the best combined performance, then submit it.
The description says the scoring follows Shapley values, a game-theory method for fairly splitting credit among team members. That means testing every subset of examples, which grows fast (2 to the power of K runs for K examples). The filing suggests batching the runs into one call and using a quantized version of the model (a shrunken copy that needs less computing power) to cut cost.
Other claims extend this to written instructions inside a prompt, display the scores as a chart, and add a plain-language recommendation. Success can be measured by translation accuracy or by how many test cases translated code passes.
… each attribution score reflects a marginal contribution of a corresponding context example.
Translation: A score is calculated to measure how much each individual example helps or hurts the AI.
Why this matters if your AI answers keep missing
If you have ever fed an AI sample answers and watched it get worse, this targets that frustration. Instead of guessing which example to swap, you would get a ranked list or chart showing what is helping and what is dragging results down, plus an automatically fixed prompt. For teams using AI to translate code or text, that could save a lot of trial and error.
The catch is cost. Testing many combinations means many extra runs of the AI, which is why the filing talks about batching and a cheaper model for the tests. It is a patent application, so there is no product here, only a described method.
IBM's 65th filing we've tracked since May in the AI safety guardrails space builds on earlier applications like one on good-and-bad answer training and one on removing staff from sensitive calls.
For anyone who writes prompts, the benefit is easy to picture: you stop guessing. A tool that tells you example number three is the problem saves you an afternoon of fiddling.
You would notice it when an AI given several samples returns uneven results. The filing's example, with scores of +6, +4 and -2, shows the fix: remove the one negative sample and the total improves.
The price is extra computing. Trying every mix of K examples takes 2 to the power of K runs, so the payoff goes to people with a handful of examples and a task where accuracy matters, such as code translation. Casual chatbot users would probably never see it.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time IBM patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
6 drawing sheets from US 2026/0330144 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in