AI Patent Teaches Machines to Fill In Missing Audio and Visual Gaps

By Patentlyze Team · Updated Jul 21, 2026

When you cover part of a photo, your brain uses context from the rest of the image to guess what's hidden. Google is patenting a way to train AI systems to do the same thing, but across two completely different types of input at the same time, like video frames and audio.

How Google's multimodal AI fills in what it can't see

Imagine watching a movie with part of the screen blacked out. If the soundtrack is still playing, you can make a pretty good guess about what's happening in the hidden section. Google's patent describes a training method that teaches AI models to do exactly this kind of cross-context guessing.

The system works by giving the AI two streams of information at once, for example, images and sound, or video and text. It hides certain pieces of one stream, then asks the AI to reconstruct what's missing using clues from both streams. Over many rounds of training, the AI gets better at understanding how the two streams relate to each other.

The end result is an AI model that has genuinely learned the relationship between different types of input, not just pattern-matched within one. That kind of cross-modal understanding is what you need for things like video search, automatic captioning, or AI that can describe a scene from both what it sees and what it hears.

How the encoder and decoder split the work across inputs

The patent describes a training framework for what researchers call a multimodal autoencoder. An autoencoder is a type of AI model that learns by compressing information into a compact form and then reconstructing it. The "multimodal" part means it handles more than one type of data at the same time, such as video frames alongside audio waveforms.

Here's how the training loop works:

• Each training example contains two sets of data tokens (small chunks of information representing each input type). Some tokens are intentionally hidden, or "masked."
• An encoder reads the visible tokens from both input types and produces a compressed representation focused on the first type.
• A decoder then tries to reconstruct the hidden tokens for that first type, using the encoder's output as its only guide.
• The system measures how wrong the reconstruction is (this is called the loss) and adjusts the model's internal settings to do better next time.

The key insight is that the encoder must process both input types simultaneously. That forces it to learn relationships between them. If audio cues help predict missing video details, the model will figure that out on its own through repeated training.

What this means for AI that understands video and sound together

Models that genuinely understand multiple input types together, rather than processing each separately and stitching results together, tend to perform better on real-world tasks. Video search, automatic meeting transcription, accessibility tools for the hearing-impaired, and AI-assisted video editing all benefit from a model that treats sound and image as two sides of the same story.

For Google specifically, this kind of training approach feeds directly into products that deal with video at enormous scale, think YouTube search, Google Lens, or any future version of Gemini that needs to reason about what it sees and hears at the same time. The patent doesn't name a product, but the research direction is clear.

Editorial commentary on a publicly published patent application. Not legal advice.