Sony Patent Separates Real Objects From Screen Recordings for Independent Editing
When a camera films a person standing in front of a screen, the screen content and the person get baked into one image. Sony's new patent describes a system that keeps them cleanly separated so each can be edited or replaced independently.
What Sony's screen-and-object separation system actually does
Imagine a news anchor standing in front of a giant video wall. When you film that scene, the camera records everything as one flat image: the person, the wall, and the graphics on it, all mixed together. Swapping out the wall graphic or changing what the anchor is wearing after the fact normally means complex, error-prone editing work.
Sony's patent describes a system that uses a "mask" (essentially a precise digital cutout) to tell the camera footage which pixels belong to the screen and which belong to the object or person in front of it. Once those two layers are separated, each one can be replaced or combined with new imagery generated from 3D scene data.
In practice, that means you could swap the screen behind a presenter with a completely different image, or wrap new 3D graphics around the foreground object, all without the two ever visually contaminating each other. Think of it as automatic green-screen, but working on a real display instead of a painted wall.
How the mask splits display pixels from object pixels
The patent covers an information processing device that takes a single camera frame capturing both a display device (like an LED video wall or monitor) and a physical object in front of it, then splits that frame into two distinct regions using mask information (a pixel-level map that marks which areas belong to the screen and which belong to the object).
Once separated, the system can do one or both of the following:
- Replace the display image area with a new image generated from 3D data, so the screen content can be completely swapped post-capture without affecting the object in front of it.
- Composite the object image area with a second image also derived from 3D data, where that second image spatially corresponds to where the screen was, so the two align correctly in 3D space.
The key detail is that both replacement images are rendered from the same 3D scene data, which keeps lighting, perspective, and spatial relationships consistent between the swapped screen content and whatever is composited onto the foreground object. The mask is what makes clean separation possible in the first place, preventing screen glow or reflected color from muddying the object layer or vice versa.
What this means for virtual production and mixed-media video
Virtual production stages, where actors perform in front of large LED walls displaying real-time 3D environments, are now common on major film and TV sets. The hard problem has always been post-capture flexibility: once you've filmed the actor against the wall, changing the background means wrestling with color spill and perspective mismatches. Sony's approach, tying both the screen replacement and the foreground composite to the same underlying 3D data, keeps everything geometrically and visually consistent without requiring a traditional green screen.
For Sony specifically, this connects neatly to its position in both professional broadcast cameras and large-format LED display systems (its Crystal LED line). A patent like this suggests Sony is thinking about tighter integration between its display hardware and its camera and production software tools, potentially simplifying workflows that currently require multiple vendors and expensive manual cleanup.
This is a focused, technically specific patent with a clear use case in virtual production and broadcast, not a vague platform play. Sony already sells both the cameras and the LED walls that would use something like this, so the vertical integration angle is real. It won't make headlines outside production circles, but it's a sensible piece of IP for a company that actually makes all the hardware in the chain.
The drawings
23 drawing sheets from US 2026/0222669 A1 · click any drawing to enlarge
Which company should we read for you?
We track 17 companies here. Pro is the same weekly breakdown for any company you choose, delivered privately. Type a name and we'll scope it and send you a quote.
Get one Big Tech patent every Sunday
Plain English, intelligent commentary, no hype. Free.
Editorial commentary on a publicly published patent application. Not legal advice.