Microsoft Files Patent to Rewrite What Speakers Say in Live Video, Lips Included
Microsoft has filed for a system that fixes a speaker's words while you watch, then redraws their mouth so the edit doesn't show. It could cut the 'ums' or translate a foreign language on the fly.
What Microsoft's live speech-rewriting patent does
Ever sat through a presentation where the speaker says 'um' and 'you know' every few seconds? Microsoft has filed a patent application for software that would clean that up while the talk is still happening, right on your own device.
An AI would listen to the live audio and flag words you would rather not hear, using a profile you set up (things like your age, the languages you speak, or where you live). Then it swaps in better words and redraws the speaker's mouth in the video so the lips match. Short blended transitions hide the seams.
The same setup could work as a translator. If someone speaks a language that isn't in your profile, you would hear and see them speaking yours. The delay would be a few seconds, not a full edit job afterward.
… identifying, from a plurality of spoken words as scanned from the live audio recording, one or more spoken words to modify according to a first set of parameters and based on a profile of a user, using the AI agent …
Translation: The system uses AI to figure out which words to change based on custom viewer rules.
How the AI swaps words and redraws the lips
The claims describe a system that watches a live video feed and its audio track while people are speaking. An AI agent (software that makes decisions on its own) listens to the words and picks out the ones to change, based on a set of rules and on a profile of the user.
Then a generative AI system (AI that creates new text, audio or images) does two jobs in 'near-real-time,' which the filing defines as within a few seconds, roughly 1 to 10:
- It replaces the chosen words with new ones, in both the audio and the video.
- It reworks the audio and video just before and after the swap, so the join is smooth.
The filing lists what could get swapped: filler words like 'um,' slang, mumbled or mispronounced words, offensive words, and background noise. Replacement words can come from an internet search, and the filing also floats instant fact-checking.
A second independent claim covers translation. Words in a language missing from your profile are swapped for translated words, and the video is redrawn, lips included, to match. It could run on AR glasses, a phone, a tablet or a video camera, using the chips on the device. One dependent claim keeps the original recording as 'ground truth.'
What this means for live talks and translation
If this ever worked well, you could watch a live lecture, a concert or a meeting in another language and see the speaker's lips match what you hear. You could also get a cleaner feed that skips the 'ums' without a video editor. The filing names AR glasses and phones as the devices.
The catch is trust. The same tool can change what a speaker appears to say, and the filing mentions swapping in facts from a search and removing offensive words, with only one dependent claim saving the original recording. The examiner's first review rejected the claims as filed, which is routine and usually followed by amendments.
This is the sixth Microsoft filing we've tracked in AI photo editing race since July, following one on frequency-aware training and one on content-aware photo cleanup.
Claim 1 is wide. It covers any system that scans live video and audio of people speaking, picks words to change using a user's profile, has generative AI swap them in both sound and picture, smooths the seams, and shows the result a few seconds later. It never says why the words get changed, so filler cleanup, politeness filtering and fact swaps all fit under it.
The translation idea sits in a separate claim, number 16, and that one is much narrower. It requires words in a language missing from your profile and lips redrawn to match. If both were granted, anyone building a live feed that swaps spoken words would run into claim 1, and a live translator with redrawn lips would also face claim 16.
Claim 1 also says nothing about telling viewers an edit happened. Only a later claim keeps the original recording. A claim this broad invites narrowing, and the first rejection on file makes that a live possibility.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Microsoft patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
11 drawing sheets from US 2026/0341309 A1 · click any drawing to enlarge
Show the other 3 sheets
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in