Adobe Files Patent for an AI That Describes Exactly the Part of a Photo You Select
Ask most AI tools to describe a photo and you get the whole scene. Adobe's patent application lets you point at one object and get a description of only that.
What Adobe's region-aware photo captioner does
Every time you ask an AI to describe a photo, you tend to get a take on the whole picture, even when you only care about one thing in it, like the vase on the table. Adobe's patent application is about that gap. You mark the part of the image you care about, type a request, and the AI describes that part.
The marking is an exact cutout of the object, not a loose box drawn around it. The filing says boxes are too imprecise and tend to produce vague or overly broad descriptions. You can also ask for different depths: a detailed rundown of the object's color, material and shape, a description of how it sits next to other things, or a summary of the whole scene with your selection in mind.
In the filing's example, a vase is described down to its green glass and the pink flowers inside, plus where it sits compared with the wine glasses and the fruit tray.
receiving, by at least one processing device, a text prompt and a digital image with a binary mask indicating a region of interest in the digital image …
Translation: The software takes your text request and a picture with a highlighted cutout area.
How the AI reads your selection and the photo
The system takes three inputs: a photo, a binary mask (a black-and-white cutout where white marks the pixels you selected), and a text prompt asking for a description.
- Mask-aware encoder: a small 336-by-336-pixel copy of the photo and the mask are each turned into numbers (called embeddings), then added together so the model knows which patches are the selected ones.
- Two high-resolution encoders: a sharper 1024-by-1024 version is read by two different image-reading networks, one for fine visual detail and one for structure and layout. The filing names ConvNeXT and the Segment Anything Model's encoder as examples.
- Fusion and adapter: the three sets of features are merged into one map, then translated by an adapter layer into a form the large language model (the text-writing AI behind chatbots) can read.
The language model then writes the caption the prompt asks for: a detailed attribute-aware description of the object, a dense caption that places it among neighboring objects, or a global caption of the whole scene.
The filing trains the model in three stages, using a set of 5,392 stock images with 14,590 labeled objects. In its tests, the model scored 56.84 on regional captioning judged by GPT-4, against 50.26 for the next best model listed. Swapping exact cutouts for boxes or outlines made results worse.
… generating captions for a digital image at multiple granularity levels utilizing a mask-aware vision-language captioning model …
Translation: It writes detailed descriptions of the selected photo part using specialized artificial intelligence.
Why describing one object could help editors
If you work with piles of photos, a description of one specific object, covering its color, material, shape and position, is more useful than a one-line caption of the whole picture. The filing says the editing system also lets you search for and view images and their captions, so richer captions could make a big library easier to search.
The filing names Adobe Firefly and Photoshop as software the system could work with, which points toward apps where you already select objects and could then ask what you've selected. It is a patent application, so none of this is a promised feature. It does show where Adobe is taking AI that reads images: from whole pictures to the exact piece you are working on.
Adobe's 44th filing we've tracked since May in the AI photo editing race adds to a run that includes work on room lighting from one photo and teaching video AI to follow objects.
This one sits close to a shippable feature because it needs no new hardware. Everything in the filing is software: a model, a photo, a selection and a text request. The filing also reports test scores and describes training on thousands of stock images, so the research stage looks finished.
The shortest route to a product is a describe-this-selection option inside an editing app, since selecting objects is already something people do in those apps. The filing names Adobe Firefly and Photoshop as places it could run. The open questions are speed and cost: several neural networks, including a large language model, have to run for every request. The filing describes both a server doing the work and the model being downloaded to your own device.
Verdict: a practical upgrade for anyone who handles large photo libraries, and an easier AI feature to turn into a button than most.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Adobe patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
10 drawing sheets from US 2026/0321041 A1 · click any drawing to enlarge
Show the other 2 sheets
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in