Sony Files Patent to Cut Actors Cleanly Out of LED Wall Footage
Modern film sets surround actors with giant LED screens showing digital scenery, but the lights in the room change how that scenery looks on camera. Sony has filed a patent for software that learns those shifts so it can still trace the actor's outline.
What Sony's LED wall cutout patent actually does
You're standing in front of a huge LED screen on a film set, acting out a scene while a digital forest glows behind you. A camera films you and the screen together. Later, an editor wants to swap that forest for a desert, which means cutting you out of the footage cleanly.
That sounds easy, because the computer already knows what the screen was showing. The catch is that studio lights tint and brighten the screen, so what the camera saw doesn't match what the computer sent. Software that compares the two gets confused, and your outline comes out ragged.
Sony's filing describes a fix. Before filming, the system looks at the empty screen under the studio's lights and learns how the lighting changes the picture. Once you step in, it can ignore that shift and find you much more accurately.
… infers a region of a physical object in a captured image, on a basis of a non-extraction-target image displayed or projected in an imaging space and the captured image in which the non-extraction-target image and the physical object present in the imaging space are captured …
Translation: It figures out where the actor is by comparing the camera footage with what the LED wall was displaying.
How Sony's software learns the studio's lighting
The patent covers software that infers the region of a physical object (an actor, say) in a captured image. It does this using two inputs: the image shown or projected behind the object, and the camera's footage of both together. The key condition in the claims is that the background's appearance from the camera's position and angle is known, which is true when a rendering engine draws the scene for that exact camera view.
The output is a mask image, a black and white map marking which pixels belong to the actor. A neural network (a trained pattern-finding program) produces it.
Training happens in three stages:
- Preliminary learning: the network starts with an existing background-matting method that works well under fixed lighting.
- Calibration: the camera films the empty screen, sometimes while the lighting is changed. The network is then tuned to output a mask of all zeros, meaning it learns that any difference it sees is just lighting, not a person.
- Actual shooting: the network takes the rendered image and the live footage and outputs the actor's mask, possibly in real time.
The filing also describes feeding in lighting details such as the position, type and intensity of lights. A second network can restyle a new image, such as an AR object, so it matches the studio's lighting before it is layered onto the footage. The filing says this could let several cameras share one LED wall, and that it also works with projected images.
Changes in the lighting environment are absorbed, and the extraction target region is inferred with high accuracy.
Translation: The system handles shifting stage lighting so it can cleanly separate the actor from the background.
Why film studios could care about Sony's cutout trick
For film and TV crews, cutting actors out of footage is routine post-production work, and a bad cutout means hours of manual cleanup. Software that holds up under changing studio lights could trim that cost. The filing itself says better masks make later editing more efficient and cheaper.
The approach also covers adding computer-made objects that match the studio's lighting, and using multiple cameras on one wall. The application is early: the examiner's first review rejected the claims as filed, which is routine and usually followed by amendments. It describes no product, though Sony sells both cameras and studio gear.
Sony's 19th patent we've tracked since July in the AI photo editing race builds on earlier applications like adjusting settings post-shot and cutting subjects with dual AI.
The problem here is real but narrow. Anyone who has watched a badly cut-out actor shimmer against a fake background knows how distracting it looks, and fixing it by hand costs film crews real time and money. The filing does not claim to solve filmmaking, only the gap between what a screen was told to show and what the camera actually saw.
The fix fits the size of that problem. Studios can film the empty screen in the lights they already use, and a short calibration step teaches the software to ignore the color shift. That is a small, practical ask for a tedious daily headache.
The audience is small: virtual production stages, not everyday filmmakers or phone users. Within that world, though, this is a sensible idea aimed at a costly chore.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Sony patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
21 drawing sheets from US 2026/0332721 A1 · click any drawing to enlarge
Show the other 13 sheets
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in