Samsung Files Patent for an AI That Can Both Describe and Search Your Photos
AI models that chat about photos are good at describing a picture but clumsy at finding one. This Samsung filing trains a single model to do both, using short captions to learn the gist and long ones to learn the details.
What Samsung's photo-describing AI learns to do
Imagine typing 'my dog running in a park' into your phone's photo gallery and getting the right shots back from thousands of pictures, not just any photo with grass in it. That sounds simple, but many AI systems stumble on it. They catch the words 'dog' and 'park' but miss who is doing what.
Samsung's researchers describe a fix: train one AI model with two kinds of homework. Photos paired with short captions teach it the big picture, that this image and these few words belong together. Photos paired with long, detailed captions teach it to notice fine points, like which object sits where. Both lessons happen at the same time.
The result, according to the filing, is a model that can still chat about a photo and can also find the right photo when you search.
… obtaining a training dataset comprising input images, wherein each input image has at least one text caption and wherein each text caption is either a short caption or a long caption; and training the pre-trained LVLM to perform discriminative image-based tasks …
Translation: The AI learns using photo collections that include both brief notes and detailed descriptions.
How Samsung trains with short and long captions
The starting point is an existing large vision-language model: a chat-style AI built from a vision encoder (the part that looks at pictures) and a large language model (the part that reads and writes text). Models like this are trained to generate text, so they describe images well but are weaker at discriminating, meaning picking the right image out of a pile.
Samsung's recipe trains it on photos with captions of two lengths, using two lessons at once:
- Short captions (up to about 30 tokens, which are roughly word-sized chunks): the photo and its caption each go through the model with a prompt like 'in one word, describe the image.' The model's final output becomes a numerical fingerprint (an embedding) for each, and training pulls matching fingerprints closer together. This is called a contrastive loss.
- Long captions (100 or more tokens): the model gets only the photo plus 'describe the image in detail' and must write the caption one token at a time. It is scored on how closely each token matches the real caption (an autoregressive loss).
The filing says the split matters: using the contrastive lesson on long captions fails because the task becomes so easy the model stops learning within a few hundred steps. Both lessons are minimized together, and small low-rank adapters (a thin layer of extra adjustable settings added on top of a frozen model) keep the training cost down.
generating, using the pre-trained LVLM, an image embedding for the input image and a text embedding for the short caption and minimising a contrastive training loss using the image embedding and the text embedding.
Translation: It matches photos with quick tags so you can easily search for your pictures later.
Why this matters for searching your photo gallery
For you, the most direct payoff is search. The filing describes typing something like 'find pictures of my dog running in a park' and getting matching shots from a gallery with thousands of images. It also describes reading a long photo description aloud, which could help people with low vision, and a chatbot that first fetches the right documents or images and then answers questions about them.
The appeal for a phone maker is one model doing two jobs, instead of a separate chat AI and a separate search AI. The filing lists smartphones, smart TVs, tablets, robots and cars as possible hosts. It is a patent application, though, and it names no specific Samsung product.
Samsung's 23rd filing in our Language AI coverage since May adds to a run that includes a two-stage reading method and auto-finding video highlights.
Searching a phone full of photos is a daily annoyance, and a common AI approach has a known flaw: it treats a sentence like a bag of words, so 'man bites dog' and 'dog bites man' look almost the same. That is a real problem for anyone who has ever scrolled for ten minutes to find one picture.
Samsung's answer fits the size of it. Instead of building a new system from scratch, it takes an existing photo-chat model and teaches it a second skill with small add-on adjustments, which keeps the training bill down. The filing's own tests show gains of roughly 5 to 7 points on standard photo-search benchmarks, and it beat a much larger model on one of them.
The problem is big and the fix matches it. The proof so far is academic: those results come from research datasets, and the filing never says how this would perform on a real phone gallery.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Samsung patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
23 drawing sheets from US 2026/0320989 A1 · click any drawing to enlarge
Show the other 15 sheets
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in