Sony Files Patent for Matching Up 3D Characters Using Flat Snapshots of Each One
Sony has a patent application for lining up two 3D characters by taking a flat picture of each and letting an AI find the matching spots. The twist is the trip back: every match is traced onto the exact spot on the 3D model.
What Sony's flat-snapshot matching actually does
Ever tried to match a rough sketch of a face with a detailed portrait of the same person? You'd look for the eyes, the mouth, the chin, and tell yourself "these go together." Computers find that surprisingly hard when both faces are 3D models built from thousands of tiny points.
Sony's patent application sidesteps the problem. Instead of comparing the two 3D models directly, it takes a flat snapshot of each, like photographing two statues. An AI trained on ordinary flat pictures, where training material is plentiful, spots which parts match, say left eye to left eye. The system then traces each match on the picture back to the exact spot on the 3D model.
Why bother? The filing says artists now do this matching by hand when they turn video into 3D animation. Automating it could let a simple animated face drive a much more detailed one.
… forwarding, to a featuring matching neural network, a network input comprising: (i) the two-dimensional source image, and (ii) the two-dimensional target image …
Translation: The system feeds both flat pictures into an artificial intelligence network to compare them.
How Sony turns 2D matches back into 3D points
The core idea is a round trip: 3D to 2D and back to 3D. The system receives two 3D meshes (digital shapes built from a net of tiny triangles), a source and a target. In the filing's main example the target is the more detailed one, and both are faces.
It renders each mesh into an ordinary 2D image, using a virtual camera and a virtual light, much like a studio photo. Both images go into a feature matching neural network (an AI model that finds the same feature in two pictures). The network returns pairs of matching spots, such as eye to eye, and can add a confidence score for each pair.
Then comes the step the claim hinges on: inverse rasterizing. While rendering, the system records which triangle each pixel came from. Run backward, that record turns every matched pixel into points on the 3D model.
The filing also describes optional extras:
- Filters that throw out likely mismatches before the trip back to 3D
- Repeating the process from several sides (front, back, top and so on) so hidden spots still get matched
Why this could speed up 3D animation work
For animation teams, this targets a slow, manual job. The filing lists uses such as turning old hand-drawn cartoons into 3D, turning 2D game cutscenes into 3D assets, converting older 2D content for virtual reality, and converting videos shot on a phone. If you've ever wondered why remaking old animation in 3D is so costly, hand-matching points between models is part of the answer.
This patent covers only the matching stage of a three-part pipeline. The other two stages build a rough 3D animation from video and then transfer that motion onto the detailed model. The filing notes that matching happens once per pair of models, which is how it describes keeping live video conversion faster than the frame rate.
Sony's 23rd filing we've tracked in AI simulation since May follows one cutting cloud-game lag and one converting video to 3D.
Claim 1 is short, and that makes it wide. It asks for two 3D models, a flat picture of each, an AI that pairs up spots between the pictures, and a step that carries those pairs back onto the models.
The claim never mentions faces, animation, video, or one model being more detailed than the other. Those details show up only in later claims and the description. Claim 2 leaves the AI wide open too, listing five network families. As written, the claim reaches any tool that matches two 3D models through this snapshot, match and trace-back loop, whatever AI sits in the middle.
The idea itself is sensible. Matching spots in flat pictures is an area with plenty of training material, according to the filing, so the claim borrows that strength for 3D work. Whether it matters outside animation depends on what examiners make of that breadth.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Sony patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
31 drawing sheets from US 2026/0350921 A1 · click any drawing to enlarge
Show the other 23 sheets
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in