Adobe Files Patent to Make Lips Match Overlaid Audio Without Losing the Speaker's Style
Dubbed video has a familiar flaw: the new audio plays while the speaker's mouth is still shaping the old words. Adobe's filing describes software that redraws the mouth to fit the new audio while keeping the person's own way of talking.
What Adobe's lip-sync patent does to a face
Imagine a film dubbed into Spanish. The audio is great, but the actor's mouth is still shaping English words, and it looks off. Worse, the little quirks that make that actor recognizable, like how they smile mid-sentence or round their vowels, get lost.
Adobe's patent application describes software that redraws the mouth area so the lips match whatever audio you give it. It first studies how the person talks in the video, then builds a plan for their mouth movements in that same style. A second model paints the result onto the face while keeping the person's identity and head position.
The filing says the system needs no extra training on a given person's video, so it should work on faces it has never seen, including ones with glasses, wrinkles or beards.
… generating, using an audio-to-speech-articulation model, speech articulation features based on the input prompt, wherein the speech articulation features represent a facial expression corresponding to the speech content; …
Translation: The system figures out how the mouth and face should move to match the spoken words.
How Adobe turns audio into mouth shapes
The system has two main parts. The first, called Audio2SpArt, takes audio (or typed text converted to audio) and turns it into speech articulation features: a compact set of numbers describing how the lips, teeth, tongue and jaw should look. A style prompt, built by matching short clips of the person's own video with their audio, tells it how this particular person speaks. The model itself is a 1D convolution network, a lightweight design that slides across the sound over time.
The second part, SpArtN, is built on a variational autoencoder (a network that squeezes images into a compact code and rebuilds them). It combines the mouth numbers with an identity code from reference photos and a pose code from the frame, using an attention step (a way for the model to decide which details matter most) before a decoder paints the new image. Stringing frames together gives a video synced to the audio.
- The identity encoder uses 4 to 8 reference images in training, and can drop to 1 to 4 at run time for speed.
- The pose encoder masks the jaw and neck using 3D face landmarks, which the filing says holds up under extreme head turns.
- A data pipeline called X-Tractor splits videos into scenes, finds who is speaking, and extracts audio features plus 68 facial key points.
The claims also cover training the audio model by comparing its predicted mouth features with ones taken from real video frames. The filing says that audio model trains in about 10 minutes on one machine.
A media generation model generates a synthetic image based on the speech articulation features. The synthetic image depicts a face with the facial expression vocalizing the speech content.
Translation: An AI image generator then creates a realistic picture of the person's face speaking those exact words.
What this means for dubbing and AI avatars
The filing lists movie dubbing, personalized advertising and interactive AI avatars as uses. In each case, the payoff is a talking face that matches the words without looking like a generic stand-in. For dubbing, that means the actor you recognize keeps their expressions in a new language.
The filing also says the data pipeline can detect audio and video sync problems that might point to earlier manipulation. That matters because the same technique that fixes a dub can put new words in someone's mouth. If tools like this reach everyday editing software, you may find yourself trusting talking-head videos a little less.
This is the 54th Adobe filing we've tracked in the AI photo editing race since May, adding to work like color edits by scene type and sharper subject edge tracing.
If you have ever winced at a dubbed show where the mouths drift out of step with the words, this filing targets exactly that. You would notice the result the moment you press play: the person on screen still looks and sounds like themselves, and the lips agree with the speech.
The style-keeping piece is the part viewers would feel most. A smile that lands mid-sentence or a certain lip shape on a vowel are small habits, yet they make a face read as a real person. Generic lip sync flattens them, and this approach tries to keep them.
The flip side lands on viewers too. A tool that works on almost any face makes it easier to produce videos of people saying things they never said. The filing mentions sync checks that could flag edited footage, but it says nothing about how that would reach ordinary people watching clips online.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Adobe patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
18 drawing sheets from US 2026/0340921 A1 · click any drawing to enlarge
Show the other 10 sheets
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in