Microsoft Files Patent for AI That Rewrites What Speakers Said in Recorded Video
Microsoft's patent application describes AI that can change the words a person says in a recorded video, then redraw their lips so the edit looks natural. It can even tailor the wording to whoever is going to watch.
What Microsoft's word-swapping video AI does
Ever sat through a recorded talk and caught yourself counting the 'ums' and 'you knows'? Scrubbing those out today takes a skilled editor and specialized software. Microsoft's patent application describes handing that job to AI.
The system listens to a recording and flags words that don't fit, such as filler words, slang, mumbles or mispronunciations. It rewrites the script with better words, then rebuilds the audio around them. It can also pick wording that suits the audience, using details like age, language, region or education level. Finally, it adjusts the video so the speaker's lips appear to say the new words.
The filing also covers a summary mode: the AI boils a long talk down to its key points, writes a short script, and generates video of the speaker saying it.
… identifying, from a plurality of spoken words as recorded in the audio file, one or more spoken words to modify according to a first set of parameters, by scanning the plurality of spoken words in the audio file, using an artificial intelligence (“AI”) agent …
Translation: An AI scans recorded speech to find specific words that need to be changed.
How the AI rewrites audio, then the lips
The first claim is a four-step pipeline. An AI agent (software that makes decisions on its own) scans the spoken words in an audio file against a set of parameters and flags the ones to change. If a script exists, it compares the script with what was actually said. If not, it first makes a transcript with speech-to-text.
Then a generative AI system (AI that creates new content) takes over:
- It rewrites the script or transcript with replacement words.
- It changes the audio to speak those words and generates audio transitions before and after, so the edit doesn't jump.
- It modifies the matching video frames, which can mean re-rendering the speaker's lips.
- It presents the result, optionally stitched together with the untouched parts.
The flagged words can include filler words, colloquial terms, offensive words and misused words. The replacements can come from an Internet or database search. A second set of parameters can cut pauses longer than a set limit (the description gives 5, 10 or 15 seconds), background noise and audio feedback. It can run on a phone, tablet, desktop or server, using NPUs (chips built for AI work), GPUs or CPUs.
A separate independent claim covers summaries. The AI picks key points, writes a summary script, and generates video and audio of the speakers saying it. Its prompt bundles the script, samples of the speakers, and an audience profile.
What Microsoft's AI editor means for viewers
If this reaches an editing tool, cleaning up a recorded talk would no longer need a skilled technician. The description lists lectures, podcasts, training videos, cooking shows and even movies as recordings it could handle. It also covers trimming long pauses and noise, or shrinking a long talk into a short summary video.
The patent office has approved this application, and once Microsoft pays the final fee it should be issued as a granted patent. The filing doesn't name a product. Because the system can tailor wording to an audience profile, two viewers could in theory get different versions of the same talk.
That makes this Microsoft's seventh filing we've tracked since July in the AI photo editing race, following one rewriting speaker lips and one on frequency-aware image training.
For anyone who has recorded a talk, the appeal is easy to see. You stumble, you say 'like' forty times, and the fix becomes a click instead of an editing session. You would notice it most in lectures, training videos and podcasts, where polish matters and nobody has an editor on staff.
The trouble starts when the AI does more than delete an 'um'. Swapping in different words and redrawing the lips means the video shows you saying something you never said. Choosing wording from a viewer's religion, ethnicity or cultural background raises the question of who decides what suits them.
Nothing in the claims requires telling viewers that words were changed. The cleanup is useful, but the labeling question decides whether people can trust what they watch.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Microsoft patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
11 drawing sheets from US 2026/0341308 A1 · click any drawing to enlarge
Show the other 3 sheets
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in