IBM Files Patent to Let Crowds Weed Out Wrong Labels on AI Training Data
Before an AI can tell a snake from an eel, someone has to label thousands of pictures. IBM's patent application has a crowd cross off the obviously wrong options first, so the expensive specialist faces a shorter list.
What IBM's crowd-vote label filter actually does
Every time an AI learns to tell a snake from an eel, a real person somewhere had to look at a photo and say which one it is. Multiply that by millions of images and you get a slow, costly chore, often handled by specialists.
IBM's patent application tries to shrink the chore. Say you have one photo and six possible labels: snake, cat, house fly, worm, eel and grub. Instead of asking one expert to pick, you ask 100 ordinary people. Any label almost nobody chooses, like the cat or the house fly, gets tossed out automatically.
What's left is a shorter list, which can go to an expert who now has fewer options to weigh. The filing also covers the reverse case: if a photo already has a label that barely gets any votes, the system swaps in a better one.
… determining, by machine logic, a rejected subset of rejected candidate label(s), the rejected subset comprising at least one candidate label of the plurality of candidate labels, with the determination being based upon each rejected candidate label being chosen in the plurality of responses with a frequency below a threshold frequency …
Translation: The system automatically throws out labels that people rarely picked.
How the vote cutoff trims the label list
The main claim covers a method for labeling training data (the examples a machine learning system learns from). The system receives a data set, such as an image, along with a list of candidate labels. A group of people each say which label best describes it.
Then the software looks at what people didn't pick. Any label chosen less often than a cutoff is rejected, and every other label is accepted. The cutoff can be an absolute count (a label needs at least N votes) or a share of responses (at least X percent of voters). In the filing's example, 100 non-expert annotators see six animal labels, the cutoff is 5 votes, and cat and house fly are dropped.
The description lays out a three-step workflow around this idea:
- Filter: the crowd crosses off unlikely labels.
- Compare: the crowd picks from what remains.
- Fix: hard cases go to experts, with the crowd's work attached.
The leftover labels can even decide which expert gets the job, such as a worm-and-eel specialist instead of a cat-and-fly one. A second independent claim covers fixing labels that already exist: if a data set's current label gets too few votes, the system picks a better candidate and replaces it, optionally by asking non-experts.
Technology for selecting a correct answer (for example, a correct label for a data set to be used in machine learning algorithms) from among a plurality of candidate answers, where the answers selected relatively infrequently by a plurality of human evaluators are cold from the full plurality of candidate answers to obtain a reduced subset of candidate answers.
Translation: Crowdsourcing helps filter out bad training labels so experts spend less time reviewing them.
Why cheaper AI training labels matter to IBM
The practical payoff is cheaper, cleaner training data. AI systems learn from labeled examples, and the filing itself says good labels are expensive to produce and that specialists are the costly part. If a crowd can cull the wrong answers first, fewer hard cases reach those specialists.
The description reports one embodiment, built for labeling the grammar of sentences, that reached 95% correct annotations while sending 13% of tasks to experts. It also says that setup flagged 3% of supposedly reliable existing labels as wrong. Those figures come from the filing's own description of a language task, and the photo example is an illustration. For you, the effect would show up indirectly, as the AI tools you use being trained on tidier data.
IBM's 42nd filing we've tracked in AI training & infrastructure since May adds to a run that includes one on factory photo defects and one on repeat-job mainframe AI.
Labeling is the dull bottleneck behind a lot of AI. Before a model can learn what a horse or a tumor looks like, people have to tag huge piles of examples, and skilled people cost real money. A cheap way to cut the specialist's workload goes after a genuine expense.
The fix itself is small: count the votes, drop any option under a cutoff, and hand the shorter list to an expert. The filing's best numbers, 95% correct with 13% of tasks reaching experts, come from a system that labels sentences. The animal-photo example only illustrates the idea.
The open question is a crowd that is confidently wrong. In the filing's own example, four look-alike labels each drew roughly a quarter of the votes, so the cutoff would remove little there. Still, pairing a cheap tool with a costly problem is sensible, and the claim about swapping out poorly supported existing labels could catch mistakes in data people already trust.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time IBM patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
5 drawing sheets from US 2026/0320157 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in