Get the newsletter
Google · AI / ML · Filed Feb 13, 2025 · USPTO published Aug 13, 2026 · Aug 13 release

Google Patents a System That Teaches AI to Search Across Text, Images, and Video at Once

Google has filed a patent for a system that converts text, images, and video into a common numerical format, letting AI compare and retrieve them as if they were the same type of data. That means you could search with a photo and get back a video clip that matches its meaning.

Fashion image search results ranked by similarity scores from the cross-modality AI system. Drawing from patent filing US 2026/0236522 A1.
Fashion image search results ranked by similarity scores from the cross-modality AI system.All 6 drawings ↓
Publication number
US 2026/0236522 A1(US20260236522A1)
Applicant
Google LLC
Filed · Published
Feb 13, 2025 · Aug 13, 2026
Inventors
Xi Cheng, Haiyang Qi, Jiashang Liu, Wen Zhang, Seyed Omid Fatemieh
Status when we published
Waiting for an examiner
Document
20 claims
US classification
707/769
Examiner
PHAN, TUANKHANH D (Art Unit 2154)

How Google's cross-media AI search actually works

You're searching a company's media library for footage of a product launch, but all you have is a written description and a single still photo. Normally, a search engine would struggle because the photo and the text are different formats entirely.

What Google's patent describes is a system that converts all of those formats, text, photos, and video, into the same kind of numerical fingerprint. Once everything is in that shared format, the AI can find relationships between them: a sentence can point to a matching video clip, or a photo can surface similar images even if no one ever typed a caption for them.

This has practical uses for things like product recommendations, finding duplicate records across a database, and building search tools that work across mixed media. It's infrastructure-level work, designed to plug into Google's cloud data warehouse products rather than a consumer app you'd open on your phone.

From the filing · CLAIM 1
… the data embeddings configure the semantic space to include relationships between the input text and the one or more of input image data and input video data.

Translation: The system maps text, images, and video into a shared digital space so the computer understands how they relate to each other.

How the engine maps text, images, and video into one space

The core idea here is called multimodal embedding. An embedding is a way of representing a piece of data, a word, a photo, a video frame, as a long list of numbers that encodes its meaning. The trick is that two things with similar meanings end up with similar numbers, even if one is text and the other is an image.

Google's patent describes a generation engine that:

  • Accepts input in multiple formats at once: text, images, and video
  • Selects the appropriate machine learning models for each format
  • Converts all of them into embeddings that share a single semantic space (meaning: a coordinate system where closeness equals similarity in meaning)

Once everything lives in that shared space, a query written in plain English can return a video clip that matches its meaning, even though no human ever manually linked them. This is sometimes called cross-modality retrieval.

The patent also specifies that these embeddings can be used to train downstream ML models for tasks like classification (sorting things into categories), regression (predicting a value), or clustering (grouping similar items). And the output is designed to feed directly into serverless cloud data warehouses, which means it's built to work at scale without a team managing the underlying servers.

From the filing · THE ABSTRACT
This supports a variety of application tasks such cross-modality retrieval (e.g., text-to-image search), entity resolution and deduplication, and user recommendations, as well as other tasks.

Translation: This technology helps search engines find images using text and helps databases identify duplicate files or suggest content.

What this means for Google's cloud AI search tools

For anyone using Google's cloud tools to manage large libraries of mixed media, product photos, support videos, user-generated content, this kind of system could make search dramatically more useful without requiring manual tagging of every file. The payoff is the absence of a frustrating failure: spending ten minutes writing precise keyword queries and still not finding the video you know exists.

The embedding approach also matters for deduplication and entity resolution, two unglamorous but expensive problems. If you have a database where the same customer appears under slightly different names, or the same product photo is stored in three formats, a shared semantic space can catch those matches automatically. Google's cloud AI tools, including BigQuery and Vertex AI, are the obvious home for this technology, and new Big Tech patents in the AI infrastructure space consistently show Google pushing embedding capabilities deeper into its data warehouse stack.

Source: US 2026/0236522 A1 at the USPTO (PDF). Full patent text and figures are in the official publication.

Get our take in your Top Stories

Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Google patent news breaks.

There are more where this came from

We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.

The drawings · 6 sheets

6 drawing sheets from US 2026/0236522 A1 · click any drawing to enlarge

Joint embedding vector space graph, drawing sheet 1 from US 2026/0236522 A1
Sheet 1 Joint embedding vector space graph
Multimodal embedding generation workflow, drawing sheet 2 from US 2026/0236522 A1
Sheet 2 Multimodal embedding generation workflow
Ranked cross-modality search results, drawing sheet 3 from US 2026/0236522 A1
Sheet 3 Ranked cross-modality search results
Embedding generation pipeline architecture, drawing sheet 4 from US 2026/0236522 A1
Sheet 4 Embedding generation pipeline architecture
Model breakdown for embedding types, drawing sheet 5 from US 2026/0236522 A1
Sheet 5 Model breakdown for embedding types
Server and client computing network, drawing sheet 6 from US 2026/0236522 A1
Sheet 6 Server and client computing network
Patent filing page