Sony Patents a System to Match Translated Speech With Character Mouth Movements

By Patentlyze Team · Updated Jul 3, 2026

Anyone who has watched a dubbed anime knows the feeling: the character's mouth keeps moving long after the English line ends. Sony is patenting a system that automatically scores how well a translated script matches a character's lip movements, before a single actor steps into the recording booth.

What Sony's lip-sync scoring system actually does

Imagine watching a dubbed movie where a character says a one-syllable word but their mouth moves for three full seconds. That mismatch is a lip-sync problem, and it has plagued dubbed animation for decades. Studios usually rely on a skilled director or editor to catch these issues by eye, which is slow and inconsistent.

Sony's patent describes a system that does this check automatically. It looks at the original dialogue, figures out the sequence of mouth shapes the character makes (each sound in speech, called a phoneme, produces a distinct mouth position), then compares those shapes to what the translated dialogue would require. The result is a similarity score that tells you how closely the translation matches the original lip movements.

The idea is to give translators and localization teams a fast, objective signal before recording even begins. A low score means the translated line will probably look awkward on screen, so the team can revise it first.

How the system compares mouth shapes across languages

The system works by breaking speech down into its smallest sound units, called phonemes (the 'p' in 'pat', the 'ah' in 'father', and so on). Each phoneme corresponds to a specific mouth shape, and those shapes are well-documented in linguistics and animation rigs.

The patent describes two sequences the system generates:

• A pre-translation phoneme sequence: the ordered list of mouth shapes the character makes in the original language.
• A post-translation phoneme sequence: the ordered list of mouth shapes required to say the translated version.

Once both sequences exist, the system computes a similarity degree (essentially a match score) by comparing the shapes position by position. A translation that happens to use sounds with similar mouth positions to the original will score high; one that requires very different shapes will score low.

The key insight is that this works purely from text. You do not need a pre-recorded audio track or a rendered animation to get a score. That makes the tool useful very early in a localization pipeline, when a translator is still drafting alternative phrasings.

What this means for dubbed anime and localized animation

For anyone who watches dubbed content, this is about picture quality in a different sense: whether what you see matches what you hear. Sony is one of the world's largest producers of dubbed animation through its Sony Pictures and Crunchyroll holdings, so an internal tool like this would have obvious commercial use at scale.

More broadly, the patent points to growing automation in the localization industry. Getting a lip-sync score automatically and cheaply could let studios evaluate dozens of translation options at once, rather than relying on a single translator's best guess. That could shorten turnaround times for global releases, which increasingly happen simultaneously across many language markets.

Editorial commentary on a publicly published patent application. Not legal advice.