Google Files Patent for Search That Takes a Photo and a Text Clue at Once
Most searches are either a photo or a sentence. Google is patenting a system that takes both at the same time, letting you say 'find me something like this picture, but in blue' and actually mean it.
How Google's combined image-and-text search works
Imagine you're shopping and you snap a photo of a chair you love, but you want the same style in a different color or material. Today's search tools usually force you to choose: upload a picture or type a description. You rarely get to combine them into one meaningful query.
This Google patent describes a system that accepts both at once. You hand it an image and a short text note (like 'but in green' or 'with a taller back'), and it figures out what you're really looking for by merging those two signals into a single search.
The trick is that the system converts the image into something that looks like text to the AI, so the visual and written parts can be compared and combined on equal footing. The result is a ranked list of images that match your combined intent, not just the picture alone.
receiving a query comprising: (i) an image depicting a scene, and (ii) a text prompt describing a context of the scene; …
Translation: The system accepts both a picture and a written description at the same time.
How the mapping network translates pixels into language
The patent describes a pipeline called composed image retrieval, which takes a query made up of two parts: an image and a text prompt. Here's how the process flows:
- A visual encoder (a neural network trained to understand images) reads the photo and produces a compact numerical representation called a visual embedding.
- A mapping neural network then translates that visual embedding into language tokens, small chunks of text-like data the same format that large language models use to process words. This is the key step: it turns a picture into something the text side of the system can understand.
- The text prompt you typed is also broken into language tokens in the standard way.
- Both sets of tokens, the image-derived ones and the text-derived ones, are fed together into a language encoder, which produces a single combined embedding representing the full query.
- That combined embedding is then used to search a large set of candidate images and rank the ones that best match your intent.
By converting the image into a token format before mixing it with text, the system avoids the usual awkwardness of forcing two very different data types to interact directly.
… processing the visual embedding of the image, using a mapping neural network, to generate one or more language tokens of the image; …
Translation: The AI translates the picture into data that computers can understand alongside text.
What this means for Google Lens and visual shopping
Google already offers image search (Google Lens) and text search, but combining them meaningfully, where the text actually modifies what the image query means, is harder than it sounds. This patent describes exactly that capability. It has obvious applications in shopping, travel (find a hotel that looks like this photo but with a pool), and creative work (find stock photos similar to this reference image but shot outdoors).
For you as a user, this could make visual search feel a lot more like talking to a person who can look at the same picture you're holding. The broader competitive angle is that similar research is being pursued across the industry, and Google staking out this specific architectural approach, mapping images into the language model's native token format, signals where the company thinks multimodal search is headed.
This is solid, well-scoped research that addresses a real limitation in image search today. The image-to-language-token mapping idea is genuinely clever because it lets a single text-based engine handle both inputs without a bespoke multimodal architecture. It's not flashy from the outside, but it's the kind of infrastructure bet that could improve Google Lens results within a product cycle or two.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Google patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
10 drawing sheets from US 2026/0228272 A1 · click any drawing to enlarge
Show the other 2 sheets
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →