IBM Patents a System for Grading AI-Written Articles Against Human-Edited Ones
How do you know if an AI wrote something good enough? IBM is patenting a method that answers that question by measuring AI-generated articles against a human-edited benchmark, using geometry to quantify the gap.
How IBM's AI article grader actually works
Imagine a magazine editor who reads every article and grades it on a handful of qualities: accuracy, clarity, tone, originality. Now imagine doing that for thousands of articles a day, all written by an AI. That's the problem IBM is trying to solve.
The idea is to take a batch of AI-generated articles and split them into two piles: ones that a human editor has touched and polished, and ones left exactly as the AI produced them. Both piles get scored across several quality categories. The human-edited pile becomes the benchmark, a kind of "good enough" standard.
To measure how close any article comes to that standard, IBM's system maps the scores onto a shape, a polygon, in a virtual multi-dimensional chart. A bigger, fuller shape means higher quality across the board. The system then compares the shapes of unedited AI articles against those of the human-polished ones to flag which outputs fall short.
How IBM builds a quality polygon from scoring data
The patent describes a quality-evaluation pipeline for content produced by generative AI models (systems like large language models that write text from a prompt). The process has several distinct stages:
- Generation and splitting: A generative AI model produces a batch of articles. Half are handed to human editors to polish; the other half are left untouched.
- Multi-dimensional scoring: Every article in both groups receives scores across a defined set of quality dimensions (think: factual accuracy, readability, tone, coherence).
- Covariance calculation: For the human-edited group, the system calculates covariance (a statistical measure of how scores across different quality dimensions tend to move together, so that related qualities are treated as linked rather than independent).
- Polygon construction: Each article's dimension scores, combined with the covariance data, are used to draw a polygon in a multi-dimensional space, a mathematical chart where each axis represents a quality dimension. A higher-quality article produces a larger, more expansive polygon.
- Threshold setting: The average polygon area for the human-edited group becomes the baseline threshold, the minimum shape size an unedited AI article must reach to be considered publication-ready.
The term "theory of mind" in the title refers to the concept of modeling another agent's perspective, here applied to gauging whether an AI's output reflects the judgment a human editor would apply.
What this means for AI-generated content at scale
As companies deploy AI to produce content at volume, the question of quality control becomes expensive and hard to answer. Human review doesn't scale, and simple word-count or readability checks miss a lot. IBM's approach offers a way to automate that editorial judgment by anchoring it to real human editing decisions, not abstract rules.
For enterprise publishers, marketing teams, or any organization pushing out AI-written material, a system like this could automatically flag articles that need a human pass before they go live. It could also give organizations a consistent, documented standard for what "good enough" means, which matters when you need to explain content quality decisions to clients or regulators.
This is a sensible, practical idea that addresses a real headache in enterprise AI adoption: how do you define and enforce a quality bar when a machine is doing the writing? The geometry-based scoring approach is clever because it captures the relationships between quality dimensions, not just the scores in isolation. Whether it holds up across diverse content types is a real open question, but the core framing is worth watching.
The drawings
9 drawing sheets from US 2026/0195229 A1 · click any drawing to enlarge
Which company should we read for you?
We track 17 companies here. Pro is the same weekly breakdown for any company you choose, delivered privately. Type a name and we'll scope it and send you a quote.
Get one Big Tech patent every Sunday
Plain English, intelligent commentary, no hype. Free.
Editorial commentary on a publicly published patent application. Not legal advice.