Sony Files Patent for a Way to Turn Filmed People Into 3D Models Without a Green Screen
Sony wants to film a person from many angles and turn them into a 3D model without hanging a green screen around them. The trick is to teach the computer the empty room first, then let it figure out what doesn't belong.
What Sony's green-screen-free 3D capture does
Today, making a 3D video of a person usually means surrounding them with a green or blue screen so a computer can tell the person apart from the backdrop. That limits what the person can wear, and the screen's color bounces onto their skin and clothes, which can leave holes in the 3D copy or green fringes in the final video.
Sony's patent application takes a different route. The system first builds a 3D model of the background alone, like a digital copy of the empty studio. Then it films the person from many angles and builds a 3D model of just the person, using the empty-room model to work out what is scenery and what is not.
The result is a cleaner 3D person, and a studio that no longer has to be wrapped in a colored screen.
… a foreground model generating unit configured to generate a foreground model that is a 3D model of a foreground, based on a foreground-background image obtained by capturing a foreground and a background from a plurality of viewpoints and a background model that is a 3D model of the background.
Translation: The system builds a 3D model of a person using multiple camera angles and a known background instead of a green screen.
How Sony separates a subject from the room
The filing describes a system with a ring of cameras (the example uses eight) placed around a subject. It captures two kinds of pictures: images of the empty background, and images of the subject standing in front of that background. Software first figures out where each camera sits and how its lens behaves (the camera parameters).
From the empty-room pictures, it builds a background model, a 3D stand-in for the set. Then it builds the foreground model, the 3D person, in a loop:
- Draw the current person model and background model from one camera's point of view.
- Compare that drawing to the real photo from that camera and measure the difference (a loss function, basically a score for how wrong the guess is).
- Adjust both models to shrink the difference, then repeat for every camera.
The background model gets updated too, so things like the subject's shadow on the floor, which weren't in the empty-room pictures, get absorbed into the background instead of the person. The claims name 3D Gaussian Splatting and NeRF (two ways of storing a scene as glowing 3D points or a learned 3D volume) as ways to build the models.
The filing also describes variants: making the background pictures from the subject pictures by blacking out the person, or importing a background model made elsewhere.
Why this could simplify Sony's 3D video studios
If you ever watch a concert, a sports replay, or a virtual class where you can move your viewpoint around a real person, this kind of capture is behind it. Dropping the green screen means studios could shoot in more ordinary spaces, and performers could wear green or shiny clothes without breaking the effect.
The filing lists uses like adding a real performer to a virtual world as an avatar, producing video of someone appearing at a scanned location, and live two-way sessions such as a teacher and student or a doctor and patient. It describes a processing pipeline, not a finished product, and says nothing about a consumer device.
Sony's 605th filing in our Sony coverage since May continues a camera privacy thread that includes face scrambling before capture and a mostly asleep camera chip.
Sony's idea here is software, not new hardware: set up cameras around an empty room, record what it looks like, then film a person and let the system subtract the room to build a three-dimensional model of just the human. Everything the approach depends on, cameras, calibration, compression, already exists in professional capture studios.
That's actually what makes this close to useful in a narrow setting. A studio that can control its own lighting and shoot the empty room first has almost everything it needs to try this today. The gap between "possible in a studio" and "works on your phone or laptop" is enormous, and the document doesn't attempt to close it.
What's missing matters too. There are no accuracy numbers, no speed measurements, and no comparison against cheaper alternatives like green screen. Until someone runs those tests, the real question of whether the output looks good enough to ship stays open.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Sony patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
11 drawing sheets from US 2026/0320913 A1 · click any drawing to enlarge
Show the other 3 sheets
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in