Google Patents a Visual Search That Guesses What You Want Before Answering
When you take a photo of something and ask Google about it, the system currently treats every search the same way. This patent describes a version that first figures out what you're actually trying to accomplish, then answers differently depending on the answer.
What Google's intent-reading photo search actually does
Imagine you photograph a pair of shoes. Are you trying to find out who makes them, buy a pair, or just learn what style they are? Right now, a visual search engine might give you the same kind of generic results either way. Google's patent is about changing that.
The system described here uses AI to look at your photo and make a judgment call about your likely goal before generating any response. Based on that guess, it selects a different "recipe" for how the answer should be structured, and then uses a text-generating AI to write the actual response in the right format.
So a photo of a dish at a restaurant might get a recipe-style answer, while a photo of a rash might get a symptom-explanation format, and a photo of a vintage jacket might get a shopping-focused response. The same photo search engine, different answers shaped around what you probably wanted.
How the system picks the right answer format per photo
The system works in a chain of steps:
- Image intake: You submit one or more photos as your query, no text required.
- Intent classification: A machine-learning model analyzes the image and produces an "intent determination" (a label predicting what the user is trying to do, such as identify, shop for, learn about, or compare).
- Prompt selection: Based on that intent, the system pulls a specific prompt from a stored "prompt library." Each prompt is a set of instructions telling the generative AI how to format its response.
- Visual matching: Separately, the system runs the image through a conventional visual search to find matching content items (web pages, products, images).
- Response generation: A large language model then takes the selected prompt plus the top search results and writes a final answer shaped to the predicted intent.
The key design choice is keeping intent detection and content retrieval as separate steps. The model doesn't generate the answer from scratch; it synthesizes the search results into a format the system decided in advance was the right one for what you wanted.
What this means for Google Lens and AI search
Google Lens already handles hundreds of millions of visual searches, and the gap between "here are some results" and "here is an answer" is where AI search is currently being fought over. This patent describes the plumbing for making that gap feel invisible by tailoring the response structure to the user's likely goal automatically.
For you as a user, the practical effect would be getting answers that feel less like a search results page and more like asking a knowledgeable person. For Google, it's a way to make AI-generated overviews in Search more useful without requiring you to type anything at all.
This is a genuinely interesting architectural patent because it separates intent detection from content retrieval and from response generation, treating each as a distinct problem rather than one big AI blob. That modular approach is probably the right way to build reliable AI search. It's not a flashy concept on paper, but it describes the kind of careful design work that separates useful AI features from unreliable ones.
Which company should we read for you?
We track 17 companies here. Pro is the same weekly breakdown for any company you choose, delivered privately. Type a name and we'll scope it and send you a quote.
Get one Big Tech patent every Sunday
Plain English, intelligent commentary, no hype. Free.
Editorial commentary on a publicly published patent application. Not legal advice.