AI Grading AI: Inside IBM's Model Evaluation Patent
What if instead of humans checking whether an AI gave a good answer, another AI did the checking? IBM is patenting exactly that, using a branching test structure to probe the strengths and weaknesses of any AI model it wants to assess.
How IBM's AI examiner scores another AI's work
Imagine you hire a contractor to renovate your kitchen. Instead of inspecting every cabinet yourself, you bring in a separate inspector whose only job is to test the contractor's work from every angle. IBM's patent describes something similar for AI systems.
Here, a first AI (the inspector) builds what IBM calls a "tree of thoughts", essentially a branching checklist of tasks designed to test a second AI (the contractor). The inspector AI generates the tasks, grades the answers, and produces a final report on how well the second AI performed.
If the inspector AI runs into a test it can't design on its own, it can pull in outside tools from a shared pool to help. The end result is a structured evaluation report, produced mostly without a human needing to write a single test question.
How the 'tree of thoughts' evaluation process runs
The patent describes a two-model evaluation framework. One AI model acts as an evaluator, tasked with assessing the capabilities of a second AI model that has been answering user questions.
The evaluator builds a "tree of thoughts", a concept in AI research where a model maps out multiple reasoning paths rather than committing to a single answer. In this context, each branch of the tree represents a different test task. Each node (a point in the tree) stores a solution to one of those tasks, which the evaluator either solves itself or delegates.
When the evaluator encounters a task it cannot handle on its own, it invokes a tool from a tool pool, a library of external resources (think code runners, calculators, or search functions) that can fill the gap.
- The first model generates the task list and the tree structure
- It solves most tasks itself, using external tools where needed
- Results from all nodes feed into a final evaluation report on the second model's capabilities
The approach is meant to be largely automated, reducing the need for humans to hand-craft benchmarks or review outputs one by one.
What automated AI grading means for enterprise software
Testing AI models is currently expensive and slow. Most benchmarks are static, meaning a model can be trained to score well on them without actually being capable in the real world. IBM's approach uses a live AI examiner that generates fresh tests on the fly, which is harder to game and cheaper to scale than hiring human evaluators or maintaining a fixed test suite.
For enterprise buyers, this matters because companies deploying AI in products or internal tools need reliable ways to verify those systems before trusting them with real decisions. A patent like this points toward automated quality assurance becoming a built-in feature of AI platforms rather than an afterthought.
This is a practical, unglamorous problem that every AI company is wrestling with: how do you check that your AI is actually good? IBM's answer, have a second AI generate and run the tests, is methodologically sensible, though the patent is light on specifics about how you prevent the evaluator from being equally flawed. The 'tree of thoughts' framing is borrowed from existing AI research, so IBM's novelty claim likely rests on the specific tool-invocation and report-generation pipeline.
The drawings
10 drawing sheets from US 2026/0195234 A1 · click any drawing to enlarge
Which company should we read for you?
We track 17 companies here. Pro is the same weekly breakdown for any company you choose, delivered privately. Type a name and we'll scope it and send you a quote.
Get one Big Tech patent every Sunday
Plain English, intelligent commentary, no hype. Free.
Editorial commentary on a publicly published patent application. Not legal advice.