Nvidia Patent Teaches Video AI to Copy Motion From Reference Clips
Describing motion in words is surprisingly hard. Nvidia's new patent wants to let you skip the verbal gymnastics entirely and just show an AI video generator a clip to copy the movement from.
How Nvidia's motion-copying AI video system works
Imagine you want an AI to generate a video of a dog running across a park, but you want the camera to pan and zoom exactly the way it does in a nature documentary you already have. Typing out that camera movement in words is nearly impossible. Nvidia's patent describes a way to solve that problem by letting you hand the AI a reference video and having it figure out the motion on its own.
The system watches your reference clip, extracts the movement patterns from it (things like how fast objects travel, how the camera sweeps, how the background shifts), and then adjusts the AI video generator to replicate those patterns in whatever new video it creates from your text prompt.
The result is a video that matches your written description and moves the way your example clip does, without you having to describe the motion in words at all.
How the motion feature matcher fine-tunes the diffusion model
The patent describes a text-to-video (T2V) diffusion model (an AI that generates video from a text prompt, similar to how image generators like DALL-E work, but for moving footage) that can be guided by a separate reference video.
The key insight is that rather than comparing raw pixels between the reference clip and the generated video, the system works at a higher, more abstract level called motion feature space. Think of motion features as a compressed description of "how things are moving" rather than "what exact pixels are on screen." The AI already understands motion in this abstract way from its original training, so matching at this level is more reliable.
The process works in three steps:
- A motion feature extractor pulls abstract motion patterns from both the reference video and the AI model's current internal state.
- A motion feature matcher then calculates how different those two sets of patterns are, using a mathematical measure called L2 distance (essentially, how far apart two sets of numbers are in space).
- The system fine-tunes the AI model to shrink that gap, nudging the generator toward producing motion that matches the reference clip.
The patent also notes the same motion-feature approach can be used for motion retrieval, meaning the system could search a library of videos to find clips whose motion best matches a given query.
What this means for AI video tools and creators
For anyone building AI video tools, motion control is the hard part. Text prompts are great for describing what a scene looks like, but terrible at conveying choreography, camera work, or pacing. Every major AI video platform (Sora, Runway, Kling) struggles with giving users precise control over how things move. A system that lets you supply a reference clip instead of a text description of motion would be a real practical upgrade for creators, not just a research curiosity.
For Nvidia specifically, this fits into a broader push to make its video-generation infrastructure more useful in professional and creative workflows. If this approach works at scale, you could imagine it powering tools where a filmmaker uploads a reference shot and the AI generates variations that preserve the original's camera energy and motion style.
This is a genuinely useful research direction. The observation that pixel-level comparison is the wrong abstraction for learning motion is smart, and working in the model's own internal feature space is a natural solution. Whether Nvidia ships this as a standalone product or folds it into a developer API, the underlying idea is practical enough to matter.
The drawings
11 drawing sheets from US 2026/0195587 A1 · click any drawing to enlarge
Which company should we read for you?
We track 17 companies here. Pro is the same weekly breakdown for any company you choose, delivered privately. Type a name and we'll scope it and send you a quote.
Get one Big Tech patent every Sunday
Plain English, intelligent commentary, no hype. Free.
Editorial commentary on a publicly published patent application. Not legal advice.