Sony Files Patent for Turning Ordinary Video Clips Into Detailed 3D Animation
A cartoon face smiling in an ordinary video could end up driving a highly detailed 3D face, with no artist tracing it by hand. That is the pitch of a new Sony patent application.
What Sony's video-to-3D animation patent does
You've filmed a short clip of someone smiling, then frowning, and you want a richly detailed 3D character to do the same thing. Today, that usually means artists tracking points on the face frame by frame and fitting them to a 3D model by hand. It is slow, and it is expensive.
Sony's patent application describes a three-step automatic version. First, the system builds a rough, low-detail 3D model that copies the motion in your video. Second, it figures out which parts of that rough model match which parts of a high-detail 3D model, like eyes to eyes and mouth to mouth. Third, it passes the movement across, so the detailed model performs the same moves.
The filing says this can run without human help. It also names uses like turning hand-drawn cartoons into 3D and 2D game cutscenes into 3D scenes.
… reconstructing high-fidelity three-dimensional (“3D”) animations from two-dimensional (“2D”) videos …
Translation: Turning standard flat video clips into detailed three-dimensional character models.
How Sony maps a rough 3D model onto a detailed one
The claims cover a pipeline with three stages. It starts with a source video, a sequence of frames showing some motion. The system turns that into a source animation: a series of poses of a low-detail 3D mesh (a 3D shape built from connected points, edges and small flat faces).
The filing describes the stages this way:
- Shape reconstruction: a neural network (a trained pattern-recognition program) reads each frame and outputs a list of numbers describing shape, pose and, for faces, expression. Those numbers adjust a 3D parametric model (an adjustable template) to produce the rough mesh.
- Feature matching: the system renders both the rough and the detailed mesh as ordinary 2D images, uses a second network to pair up features like eyes and chin, then maps those pairs back onto the 3D shapes (called inverse rasterizing).
- Deformation transfer: a math technique called optimal transport (a way of computing the best match between two sets of points) carries the motion from the rough mesh onto the detailed one, guided by the matched features.
The detour through 2D images is deliberate. The filing says matching in 3D is harder and has little training data, while 2D matching has plenty. Matching is typically a one-time job for each pair of meshes, so a live video stream can be converted frame by frame, faster than the video plays.
… processing the source animation, in accordance with the set of three-dimensional paired correspondences, to generate a target animation comprising a sequence of states of the three-dimensional target mesh performing the motion sequence …
Translation: Mapping the movement from a basic 3D shape onto a much more detailed 3D model.
What this could mean for cartoons and game cutscenes
If this works as described, the cost of making a detailed 3D character move like a real person, or like a cartoon, could drop sharply. The filing lists hand-drawn classic cartoons converted to 3D, old 2D game cutscenes rebuilt with free camera angles, and phone videos turned into animation.
Sony Interactive Entertainment is the PlayStation maker, so game cutscenes are the obvious fit. The filing also mentions turning older 2D content into depth-aware scenes for virtual and augmented reality. Keep in mind this is a patent application, and nothing in it says a product is coming or shows how well the method performs.
Sony's 21st filing in the AI simulation space we've tracked since May builds on earlier applications like one adjusting graphics to your actions and one tailoring games to your taste.
The big trade here is extra detail in exchange for a middleman. Everything the final animation does is copied from a rough, low-detail model first, so any mistake that rough model makes in reading your video gets carried onto the beautiful one. A bad guess about a half-hidden mouth becomes a twitchy, high-resolution mouth.
Matching through flat images is a clever shortcut, and it adds steps: render, match, then map back to 3D. The filing also admits a single view can hide features, so it repeats the process from several angles (front, back, top, bottom, left and right). That is more work, but it runs once per character pair, so the cost looks cheap.
Nothing in the filing's text shows how well it performs. Even so, the trade reads as worth it for studios sitting on piles of footage, because the alternative the filing describes is artists tracking points by hand, frame after frame.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Sony patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
31 drawing sheets from US 2026/0320891 A1 · click any drawing to enlarge
Show the other 23 sheets
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in