Microsoft Patents a Way to Clone Anyone's Voice Into a Foreign Language
Microsoft has patented an AI system that can take a short recording of someone's voice and use it to speak an entirely different language, keeping that person's vocal character intact, even if the system has never heard that person speak before.
How Microsoft's voice-cloning AI crosses language barriers
Ever wished you could hear a dubbed film where the actor's actual voice, not a stranger's, reads the translated lines? That's roughly the problem Microsoft is filing a patent to solve.
The system works by giving an AI three things: a short audio clip of the person whose voice you want to copy, a sentence written in their original language (to anchor the voice), and a sentence written in a different language (the one you want the output in). The AI figures out what makes that person's voice sound like them, then speaks the foreign text in that same voice, even though the system was never trained on that speaker specifically.
The key detail is the phrase "zero-shot": you don't have to pre-register or record hours of training audio. A single short sample is enough for the system to do its job with a voice it has never encountered.
… obtaining a speech sample comprising audio data from a target speaker, wherein the target speaker is an unseen target speaker such that no audio data from the target speaker was included in the text-to-speech training dataset …
Translation: The system can mimic a person's voice even if it has never heard them speak before.
How the model maps a voice sample to a new language
The patent describes a neural codec language model (a type of AI that treats audio the same way a text AI treats words, by breaking sound into small discrete units called "tokens") trained on bilingual pairs of speech recordings and transcripts.
At inference time, the model receives three inputs:
- A short audio clip of the target speaker, someone the model has never seen in training.
- A first-language text prompt, a written sentence in the speaker's native language, which gives the model a reference point for matching voice characteristics.
- A second-language text prompt, the actual sentence the model needs to speak out loud in a different language.
The model uses the audio clip to extract a kind of voice fingerprint, then generates audio tokens for the second-language text, conditioned on that fingerprint. The result is synthesized speech in the target language that preserves the speaker's pitch, rhythm, and timbre.
The "zero-shot" label means no fine-tuning or speaker registration is needed beforehand. The model generalizes from bilingual speech transcription pairs it saw during training to handle novel speakers at runtime, which is the genuinely hard engineering problem here.
… generating a personalized speech output based on the inputs and by at least converting the second text prompt in the second language using a synthesized voice of the target speaker based on the speech sample from the target speaker.
Translation: The technology creates a recording of the person speaking a new language using their own voice.
What this means for translation and synthetic voice tools
Voice dubbing today is expensive and slow: human voice actors re-record lines in each target language, and even high-end AI dubbing products usually substitute a generic synthetic voice rather than preserving the original speaker's character. A system that does this automatically from a short sample could change the economics of localization for video content, corporate communications, and accessibility tools.
The broader context is that voice cloning technology is moving fast across the industry, and Microsoft's filing focuses on the cross-lingual case, a harder and less-addressed variant of the problem. The approach leans on codec language models, an architecture Microsoft has invested in heavily through its research partnership with OpenAI and its own speech teams. Readers tracking where this area of AI is heading can find the latest Big Tech patents in voice synthesis and speech AI filed alongside this one.
The design's central tradeoff is accuracy versus versatility. A zero-shot system that has never heard a speaker before will almost always produce a less faithful voice clone than one fine-tuned on that speaker's recordings. Microsoft is betting that "good enough from one sample" beats "perfect but requires setup," and for most real-world use cases, that trade probably holds. The harder cost to name is on the safety side: a system that needs only a short, unseen audio clip to clone a voice is substantially easier to misuse than one requiring enrollment, and the patent says nothing about how that risk gets managed.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
10 drawing sheets from US 2026/0237379 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →