Google Files Patent for AI That Captions One Chosen Part of a Photo
Draw a box around a cat on a bicycle, and Google's system can call it 'cat' or 'white cat sitting on a black bicycle and looking down in a dark room,' depending on the length you set. One model does both jobs.
What Google's box-based photo captions do
Ever pointed at one thing in a busy photo and wished a computer could just tell you what it was? Say your picture shows a white cat on a bicycle. You draw a box around the cat, and the system writes a caption about only what's inside that box.
The twist is that you also pick how much it says. Ask for one word and you get 'cat.' Ask for about six and you get 'white cat sitting on a bicycle.' Ask for fourteen and you get 'white cat sitting on a black bicycle and looking down in a dark room.' One model handles every job from naming things to describing them.
The boxes can come from you or from a separate program that spots objects automatically, so a single photo can end up with a caption for every item in it. The filing also describes handing those box-by-box captions to a chatbot-style AI so it can answer questions about the picture.
… obtaining data specifying a location of a first bounding region within the input image that depicts a first object; generating a bounding region embedding vector that represents the location of the first bounding region; …
Translation: The system pinpoints the specific area of the image that you want to talk about.
How the photo, box and length get combined
The system hands three ingredients to a language model (the same family of AI that powers chatbots). First, an image encoder chops the photo into a grid of patches and turns each one into a list of numbers, called an embedding vector, that captures what the patch shows. Second, the box's location (its corner coordinates, for example) goes through a small linear neural network (a simple layer of math) that turns it into its own vector.
Third, and optionally, you can ask for a caption length. That request becomes a prefix token such as LENGTH-6, which is added to the input. All of these pieces are lined up into one sequence, and the language model writes the caption one word piece (a token) at a time, each new piece based on the ones before it.
Here is how the claims break down:
- Claim 1 covers the image vectors, the box vector and the language model writing the caption.
- Length control, a second box in the same photo, and an automatic object detector to find boxes are dependent claims.
- Training the encoder and language model together on next-word guessing is also a dependent claim.
For training data, the filing describes pulling short phrases out of existing photo descriptions, such as 'red disc,' and using an object detector to match each phrase to a box.
… processing the input sequence of input embedding vectors using a language model neural network to generate as output an output sequence of text tokens that represents a text caption that describes the first object depicted within the first bounding box.
Translation: An AI language model then writes a caption specifically describing just that selected part.
Why a chatbot could answer questions about photos
If you've ever asked an AI about a photo and gotten a vague answer about the whole scene, this is aimed at that. Because each object gets its own caption tied to a location, a chatbot can answer pointed questions. The filing's example is a photo of a plane: a caption reading 'the word Comfort on the side of the plane' lets the chatbot answer 'What flight is it?' with 'Comfort Airlines.' In a tennis photo, captions like 'Tennis racket' help it answer 'Wimbledon.'
The filing lists camera photos of road signs, vehicles and pedestrians, and medical scans showing organs and tumors, as possible image sources. It names no specific product, so treat this as a research-style building block for how Google's AI could describe pictures in more detail.
Google's 77th filing we've tracked since May in the AI photo editing race follows earlier applications like a video quality grader and a group-friendly zoom.
Claim 1 is short and wide. It needs a photo turned into lists of numbers, a box position turned into its own list of numbers, and a language model that writes a caption from both. Nothing in it requires a particular type of AI model, a training method, or a caption length.
The length control that makes the description fun, anywhere from one word to a full sentence, appears only in later claims. So claim 1 would reach a caption of any length written this way. If granted, that wording could cover a lot of box-based captioning built on a language model, whatever its other details.
The whole claim leans on one step: handing the box location to the language model alongside the picture. Whether it would be granted in that form is another matter. As an idea, it is sensible and useful, and the claim is ambitious.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Google patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
8 drawing sheets from US 2026/0350959 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in