Microsoft Patents a Three-Model AI System for Writing Story Captions from Photos
Writing a one-line photo caption is hard enough. Microsoft has filed a patent for a system that writes full story-style paragraphs about an image, using three separate AI models to draft, fact-check, and pick the best version.
What Microsoft's image captioning pipeline actually does
Every time a social media platform or news site shows you an image, someone (or something) has to describe it. That description tells screen readers what's in the photo, powers search results, and increasingly fills in alt-text automatically. The problem is that current AI captions tend to be either too short and generic, or confidently wrong about details.
Microsoft's system tackles this by splitting the job three ways. One AI model looks at the photo and pulls out tags, object locations, and contextual clues. A second model takes all that information and writes several full paragraph-length descriptions. A third model reads those drafts back against the original photo, cuts out any sentence that doesn't actually match what's visible, and picks the best surviving paragraph.
The system also takes a caption focus as input, meaning you can tell it whether the caption is for a social media post, a news article, or an accessibility tool, and it adjusts the tone and format accordingly.
receive, by a first vision language model, a request for a story caption of an image, the request comprising a caption focus for the image, the caption focus comprising instructions for a format of the story caption, the format being based on a platform the requested story caption is presented in …
Translation: The system takes a photo and instructions on how to format the caption for a specific website.
How the three AI models split the captioning work
The patent describes a three-stage pipeline built around two types of AI models: vision language models (VLMs, which can look at an image and produce text) and a generative language model (a text generator, similar to the kind that powers chatbots).
Stage one feeds the image to a first VLM, which extracts structured information: tags (labels like "dog," "park," "sunset"), an initial rough caption, descriptions of individual objects including their locations within the frame, and what the patent calls visual clues, which are inferred contextual details built from those tags.
Stage two passes all that structured visual information, plus a caption focus parameter (instructions about the platform and format the caption will appear in), to a generative language model. That model produces multiple candidate paragraphs, each a coherent, natural-language story description of the image.
Stage three is the quality filter. A second VLM takes each candidate paragraph and checks it sentence by sentence. It measures a similarity score between each sentence and the original image. Any sentence that falls below a defined threshold gets removed, and the paragraph is rebuilt without it. Finally, the second VLM scores all the cleaned-up candidates and returns the best one as the final output.
The design is notable for treating caption accuracy and caption quality as separate problems, solved by different models at different stages.
A second vision language model evaluates the plurality of image story caption candidates and selects a caption as the final output caption.
Translation: A second AI model reviews the options and picks the best final caption for the photo.
What this means for AI photo descriptions and accessibility
For anyone who relies on screen readers or image descriptions, captions that hallucinate details (AI confidently describing things that aren't there) are more than an annoyance; they're a real barrier. A system that actively scrubs sentences that don't match the photo could make AI-generated alt-text meaningfully more trustworthy.
Microsoft's steady investment in vision-language research shows up here in the layered, multi-model architecture. Beyond accessibility, accurate paragraph-length captions power image search, content moderation, and the kind of rich metadata that makes large photo libraries actually searchable. Whether this ends up inside Microsoft 365, Azure AI services, or Bing's image tools, the underlying problem it's solving is one that every platform dealing with large volumes of photos faces.
This is the 24th Microsoft filing we've tracked since May in our AI models working in teams, following one on shared AI memory and one on self-fixing code.
The core trade this design makes is spending a lot of compute to buy accuracy. Running three separate AI models on a single image is expensive: one to analyze, one to write, one to fact-check. That cost might be acceptable for high-stakes use cases like accessibility alt-text or medical imaging documentation, but it's hard to justify for captioning millions of vacation photos in real time.
The sentence-level similarity filter is the most interesting engineering choice here. It catches hallucinations at a granular level rather than rejecting whole captions, which is more surgical and less wasteful. The risk is that trimming sentences from a paragraph can leave something grammatically awkward or narratively incomplete, and the patent doesn't address how the system repairs the gaps.
The caption focus parameter, which lets the system adjust output based on the destination platform, is the kind of practical detail that separates a research demo from a shipping product. That detail suggests this is further along the product pipeline than a pure research filing, even if it tells us nothing about when or where it ships.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
7 drawing sheets from US 2026/0268699 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →