IBM Patents an AI That Finds Objects in Photos Using Words and Pictures Together
Most AI image tools make you choose: either describe what you're looking for in words, or show it a picture. IBM's new patent lets you do both at once, and argues that combining the two signals finds objects far more reliably than either method alone.
How IBM's dual-reference object finder actually works
Imagine you're trying to find a specific industrial part in a warehouse photo, but you can't quite put it into words, and no single reference photo is a perfect match. Today's AI tools usually force you to pick one approach: type a description or upload an example image.
IBM's patent describes a system that takes both at once. You give it a short text description of the object and a reference photo of something similar. The AI combines those two sources of information to pinpoint where the object is in a new, unseen image.
Once the system has a confident guess about where the object sits in the photo, it passes those location hints to a dedicated image-outlining model, which draws a precise border around the object. The result is a clean mask you could use to isolate, count, or label that object automatically.
How the embedding matching drives the segmentation prompt
The system works in three main steps:
- Embedding matching: The patent uses "embeddings" (compact numerical fingerprints that capture what something looks and sounds like) for three things: the target image you want to analyze, a text description of the object, and a reference image of a similar object. The system compares the target image's fingerprints against both the text and reference-image fingerprints to find which parts of the target image line up best.
- Prompt generation: Those matching positions in the image become a "segmentation prompt" (basically a set of coordinate hints that say "the object is probably here").
- Mask generation: That prompt is fed into a dedicated image segmentation model (think of it as a precise digital stencil-cutter) which draws a pixel-level outline around the object.
The key claim is that fusing text and visual references produces better location signals than either alone. Text descriptions capture abstract properties (color, function, shape); reference images capture visual specifics that are hard to put into words. Combining them narrows the search to regions that satisfy both constraints simultaneously.
What this means for industrial AI and visual search
For industrial inspection, medical imaging, or retail catalog tools, finding a specific object reliably in varied, cluttered photos is genuinely hard. Current systems that rely on text alone can be thrown off by ambiguous descriptions; systems that rely only on a reference image struggle when the new photo has different lighting or angle. IBM's approach of fusing both signals could reduce the manual labeling work required to train and deploy these tools.
For IBM specifically, this fits into its enterprise AI strategy around tools like watsonx, where customers need to process large volumes of images in manufacturing, supply chain, and document workflows without building custom datasets from scratch.
This is a focused, practical engineering patent, not a moonshot. The idea of combining text and image references for object detection isn't entirely new in the research world, but packaging it as a clean pipeline with a segmentation prompt as the handoff mechanism gives IBM something concrete to build into enterprise software. It's worth watching if you care about industrial computer vision, but it's unlikely to make headlines outside that space.
The drawings
10 drawing sheets from US 2026/0196012 A1 · click any drawing to enlarge
Which company should we read for you?
We track 17 companies here. Pro is the same weekly breakdown for any company you choose, delivered privately. Type a name and we'll scope it and send you a quote.
Get one Big Tech patent every Sunday
Plain English, intelligent commentary, no hype. Free.
Editorial commentary on a publicly published patent application. Not legal advice.