Get the newsletter
Google · AI / ML · Filed May 27, 2025 · USPTO published Nov 19, 2026 · This week's release

Google Files Patent for Teaching Image Search Which Object You Mean

Google's patent teaches search to find the dog in your photo by training on descriptions that never say 'dog.' A language model writes clues like 'the thing with four legs,' and only clues that fit exactly one object survive.

Visual examples show search prompts isolating a champagne glass, a person's hair, or a dog within an image. Drawing from patent filing US 2026/0369023 A1.
Visual examples show search prompts isolating a champagne glass, a person's hair, or a dog within an image.All 22 drawings ↓
Publication number
US 2026/0369023 A1(US20260369023A1)
Applicant
Google LLC
Filed · Published
May 27, 2025 · Nov 19, 2026
Inventors
Lav Singh, Devansh Gautam, Aishwarya Mittal, Inderpreet Singh Saggu, Manish Arora, Nitish Krishna Murthy
Status when we published
Examiner raised objections; the company can respond
Document
20 claims
US classification
382/103
Examiner
KAUR, JASPREET (Art Unit 2662)

What Google's photo-search training trick does

Ever snapped a photo of a crowded scene and typed a question about just one thing in it, like 'how to bleach?' when you meant the hair, not the shirt? Search has to guess which part of the picture you mean. The filing says existing systems can misidentify that part, or simply process the whole image.

Google's patent is about making that guess better, and the trick is in the practice material. A language model looks at a photo where one object, say a dog, is already boxed. It then writes descriptions of the dog that never use the word 'dog,' such as 'the animal' or 'the thing with four legs.' Any description that could also fit something else in the picture, like a duck, gets thrown out.

The photos and surviving descriptions then train a model to pick the right spot in a picture when you type a vague question, so the search can focus on that part of your image.

From the filing · CLAIM 1
… each of the plurality of synthetic text queries referencing the particular object depicted in the image without using a term from the object label; …

Translation: The system writes search phrases for an object without using its actual name.

How Google writes clues that never name the object

The filing covers two linked pieces. The first builds training material. The system starts with an image, a label for one object in it (say 'dog'), and a ground truth bounding box, a rectangle marking where that object sits, treated as the correct answer. A generative language model, such as a vision language model that reads images as well as text, writes several descriptions of that object without using any word from the label.

The filing's example is a yard scene with people, a dog, a duck, a house and trees. Candidate clues include 'the standing thing,' 'the animal' and 'the thing with four legs.' Each gets scored for whether it could fit some other object, and only 'the thing with four legs' survives.

The surviving clue, image and box become one training example. A multimodal region-of-interest model (software that links words to a region of a picture) scores candidate boxes against the clue, and a loss function (a measure of how far off it was) nudges it toward the right box.

The other claim group covers the search side: a photo plus typed words goes in, an object detector draws boxes around everything, the model scores each box against the words, and the search engine runs on the best match.

From the filing · THE ABSTRACT
The computing system includes multimodal region-of-interest (ROI) model, which is trained with the training example to determine one or more particular bounding boxes associated with the particular object depicted in the image.

Translation: An AI model learns to draw accurate boxes around specific items in photos.

Why this matters for Google's photo-plus-text search

If you search with a photo and a few words, you expect the search to look where you mean. The filing says current systems can misidentify the region or process the entire image, which hurts the relevance of results. A better box-picker could mean fewer follow-up searches, and cropping to the right object can also shrink what gets sent to the search engine.

The patent also leans on cheap training data: a language model writes the practice descriptions, which the filing says cuts the computing needed to gather and store training data. The examiner's first review rejected the claims as filed, which is routine and usually followed by amendments, so the final scope may be narrower. Nothing here says when, or whether, you would see it in a product.

This is the 54th Google patent we've tracked in our AI vision coverage since May, adding to work like one that scores its own guesses and one borrowing faces for group photos.

Source: US 2026/0369023 A1 at the USPTO (PDF). Full patent text and figures are in the official publication.

Get our take in your Top Stories

Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Google patent news breaks.

There are more where this came from

We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.

The drawings · 22 sheets

22 drawing sheets from US 2026/0369023 A1 · click any drawing to enlarge

Matching your text question to items in an image, drawing sheet 1 from US 2026/0369023 A1
Sheet 1 Matching your text question to items in an image
Scoring candidate objects against your query to improve search, drawing sheet 2 from US 2026/0369023 A1
Sheet 2 Scoring candidate objects against your query to improve search
How the system spots objects and picks the relevant one, drawing sheet 3 from US 2026/0369023 A1
Sheet 3 How the system spots objects and picks the relevant one
Pinpointing target items using words like bleach or champagne, drawing sheet 4 from US 2026/0369023 A1
Sheet 4 Pinpointing target items using words like bleach or champagne
Checking whether a cropped object matches your search words, drawing sheet 5 from US 2026/0369023 A1
Sheet 5 Checking whether a cropped object matches your search words
Running an image search guided by your matched text query, drawing sheet 6 from US 2026/0369023 A1
Sheet 6 Running an image search guided by your matched text query
Generating synthetic descriptions to train the object finder, drawing sheet 7 from US 2026/0369023 A1
Sheet 7 Generating synthetic descriptions to train the object finder
Training the visual search AI using labeled pictures, drawing sheet 8 from US 2026/0369023 A1
Sheet 8 Training the visual search AI using labeled pictures
Show the other 14 sheets
Steps for generating training data to teach the vision AI, drawing sheet 9 from US 2026/0369023 A1
Sheet 9 Steps for generating training data to teach the vision AI
How the vision model checks its guesses and learns, drawing sheet 10 from US 2026/0369023 A1
Sheet 10 How the vision model checks its guesses and learns
Hardware setups linking phones, glasses, and search servers, drawing sheet 11 from US 2026/0369023 A1
Sheet 11 Hardware setups linking phones, glasses, and search servers
Software parts that turn camera views into search answers, drawing sheet 12 from US 2026/0369023 A1
Sheet 12 Software parts that turn camera views into search answers
Teaching an AI model using feedback signals, drawing sheet 13 from US 2026/0369023 A1
Sheet 13 Teaching an AI model using feedback signals
A general AI model turning raw inputs into results, drawing sheet 14 from US 2026/0369023 A1
Sheet 14 A general AI model turning raw inputs into results
An AI system turning ordered data into answers, drawing sheet 15 from US 2026/0369023 A1
Sheet 15 An AI system turning ordered data into answers
Combining text, photos, and task instructions into one stream, drawing sheet 16 from US 2026/0369023 A1
Sheet 16 Combining text, photos, and task instructions into one stream
Software tools used to build and refine the search AI, drawing sheet 17 from US 2026/0369023 A1
Sheet 17 Software tools used to build and refine the search AI
Steps for training and tuning the AI using person feedback, drawing sheet 18 from US 2026/0369023 A1
Sheet 18 Steps for training and tuning the AI using person feedback
How cloud servers answer image questions from an app, drawing sheet 19 from US 2026/0369023 A1
Sheet 19 How cloud servers answer image questions from an app
Connected computers that build and run the search tools, drawing sheet 20 from US 2026/0369023 A1
Sheet 20 Connected computers that build and run the search tools
Each phone app managing its own AI tools and sensors, drawing sheet 21 from US 2026/0369023 A1
Sheet 21 Each phone app managing its own AI tools and sensors
Phone apps sharing a single central AI system instead, drawing sheet 22 from US 2026/0369023 A1
Sheet 22 Phone apps sharing a single central AI system instead
Patent filing page

Reader comments

No comments yet

Start the discussion

Real name or a handle, either is fine. Comments are read by a person before they appear, so allow a little time. Keep it about the filing.