Google Files Patent for Dubbed Videos Whose Translated Speech Stays in Sync
Dubbed videos often feel off because the translated sentence runs longer or shorter than the original. Google's patent application treats that as a math problem and solves it for the whole clip at once.
What Google's dubbing timing fix actually does
You're watching a cooking video dubbed into your language, and the narrator keeps talking long after the chef has stopped. A few seconds later the voice goes quiet while she is clearly still mid-sentence. Automatic dubbing has this problem a lot.
The cause is simple. Languages take different amounts of time to say the same thing, and translation software cares about getting the meaning right, not about how long the sentence takes. Google's patent application describes a timing step that sits between the translation and the final voice. It looks at when each original sentence started and ended, then works out how fast and when each dubbed sentence should play.
The aim is a steady, natural pace where the new voice stays close to the original timing, without sudden speed-ups or awkward gaps.
… automatically generating synthesis optimization data by solving a quadratic optimization problem formulated to synchronize the segments of automatically generated synthesized speech audio with the sequence of input images; …
Translation: Math formulas are used to match the translated voice timing with the video frames.
How the math keeps each dubbed sentence on time
The patent describes a five-step pipeline, and the timing step is the new part. A speech recognition model (software that turns spoken words into text) transcribes the original audio and splits it into segments, roughly one per sentence, each with a start and end time. A translation model then converts that text into the second language.
Next comes the timing step. Google's filing frames it as a quadratic optimization problem, a kind of math problem that searches for the lowest total "penalty" score. For each segment, the system picks a start time, an end time and a speed adjustment. Penalties grow when:
- A dubbed sentence starts or ends far from where the original did
- Speaking speed jumps sharply between neighboring sentences
- Speed strays from a target value
- Specific "pinned" words land away from the moments they should hit
The math also follows rules: all the speech must fit within the length of the original audio, sentences can't overlap, and each speed adjustment stays between a minimum and maximum (the description gives 1.0 and 2.0 as examples). Finally, a text-to-speech model (software that generates a spoken voice from text) speaks each segment as planned, and the result is combined with the original images and background audio. The filing also describes simplifying the math into a "diagonal" form to keep large jobs efficient.
… automatically generating a source transcript of the input speech, automatically generating a translated transcript in the second language by translating the source transcript, …
Translation: The system first transcribes the original speech and then translates it into the new language.
Why dubbed videos may stop feeling off-sync
If you watch videos in a language you don't speak, the pacing of the dub shapes whether the result feels watchable or distracting. A voice that lags behind, rushes, or leaves odd silences pulls your attention away from the video itself. This filing targets that annoyance directly by treating the whole clip as one timing puzzle instead of fixing each sentence in isolation.
The description says earlier approaches could produce unnatural stops or gaps, or got bogged down in too many variables. Google's pitch is a method that stays smooth and scales to long videos. It is a patent application, so it describes an idea rather than a launched feature, but it points at a problem any company offering automatic dubbing has to solve.
Google's 64th filing we've tracked in voice and speech AI since May adds to a run that includes one adding voice to text chatbots and one timing words in captions.
Claim 1 covers a full dubbing pipeline: recognize speech, translate it, fix the timing, speak the result, and put it back over the original images and background sound. Most of that is familiar. The part that sets the claim apart is a single step, solving a quadratic optimization problem to line the new speech up with the picture.
That makes the claim broad about the overall recipe and narrow about the timing method. It names no specific penalties, limits or speed ranges. Those details show up only in the later claims, which add the start, end, speed and pinned-word rules. A dubbing system that settled its timing a different way would not match that wording.
Our read: a modest, practical filing aimed at one very visible flaw in machine dubbing. It will not change how you watch overnight, but fixing awkward pacing is exactly what makes an automatic dub feel finished.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Google patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
5 drawing sheets from US 2026/0351242 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in