Sony Patents an AI Video System That Spots Newly Appeared Objects Before Rendering Goes Wrong
When an AI tries to generate smoother video frames, it often trips over one specific problem: objects that suddenly appear on screen with no prior history. Sony's new patent targets exactly that blind spot.
What Sony's appearing-pixel fix actually does for video
A character steps out from behind a wall in a video game, and for a split second the image looks smeared or ghostly. That glitch happens because the software generating the picture is trying to use information from the previous frame, where that character didn't exist yet. You end up with a brief but noticeable visual mess.
Sony's patent describes a system that catches those moments before they cause problems. It watches for pixels that show up in the current frame but had no matching pixel in the frame just before, and it flags them as "appearing pixels." Then, instead of feeding the AI that creates the finished image some stale, irrelevant data about that spot, the system swaps in a neutral placeholder value.
There's a practical twist too. If neither the previous frame nor the current frame shows any real movement in a given spot, the system decides the pixel didn't actually appear fresh; it was probably just a very slow-moving or stationary object. That check stops the system from over-correcting and flagging things it shouldn't.
… identifying, based on the (n-1)-th depth information and the n-th depth information, an appearing pixel that is among pixels of the n-th input frame and is in an area in which a whole or part of an object of the one or more objects not displayed in the (n-1)-th input frame …
Translation: The system spots pixels for newly visible objects by comparing depth data between frames.
How the system flags pixels and guards the ML model
The patent covers a pipeline for generating AI-assisted video frames from rendered 3D scenes, the kind of process used in video games, virtual reality, and real-time graphics to produce more frames per second than the hardware would normally manage.
At its core, a machine learning model takes two inputs: the current rendered frame and "auxiliary information" derived from the previous frame's internal features (the data the AI built up while analyzing that earlier image). The idea is that the previous frame's features help the model produce a cleaner, more consistent current frame.
The problem arises when an object appears in the current frame that simply wasn't there before. The previous frame has no useful feature data for that region. Feeding the model bad data causes artifacts, the visual glitches gamers know as ghosting or smearing. The system solves this by:
- Comparing depth maps (per-pixel distance-from-camera information) for the previous and current frame to find pixels where a newly visible object has appeared
- Replacing the feature data for those "appearing pixels" with a neutral preset value, so the model gets a clean signal instead of misleading history
- Cross-checking motion vectors (measures of how much each pixel moved between frames) in both the previous and current frame; if both read below set thresholds, the pixel is assumed to be stationary or slow-moving, not genuinely new, and the replacement step is skipped
That motion check is the key design judgment: it stops the system from treating a nearly-still background element as a brand-new appearing object just because depth values changed slightly.
… acquires an (n-1)-th amount of motion indicating magnitude of motion in each pixel of the (n-1)-th input frame and an n-th amount of motion indicating magnitude of motion in each pixel of the n-th input frame …
Translation: It calculates how much every pixel moves from one frame to the next.
What this means for real-time 3D rendering quality
For anyone playing a fast-moving game or using a VR headset, visual glitches around newly revealed objects are one of the more jarring experiences in modern graphics. AI-based frame generation is increasingly common in gaming hardware, so a method that specifically addresses the "object just appeared" failure case has real practical value.
The trade the design makes is worth naming: the motion-threshold check is a heuristic, a rule of thumb. Set it too high and you miss genuinely new objects; set it too low and you still feed the model bad data. Real-world scenes with complex overlapping motion could push those thresholds into awkward territory. Whether Sony tunes those values well in practice will matter more than the architecture described here.
Sony's 15th filing we've tracked since July in the GPU rendering race adds to earlier applications like one sharpening compressed visuals and one serving different streams per viewer.
The core insight, that AI frame generators need to handle "appearing pixels" differently from ordinary pixels, is correct and addresses a real failure mode. The depth-map comparison approach is a reasonable way to catch it.
The motion-check addition is where the design gets interesting, and also where it takes on risk. Two static thresholds deciding whether a pixel counts as "new" is a brittle approach in scenes with variable speed, camera cuts, or objects that move slowly into frame. A character tiptoeing around a corner could plausibly fool the system either way depending on how fast they move.
This reads as a targeted, incremental improvement to an existing AI rendering pipeline rather than a rethinking of how frame generation works. For Sony's track record in image processing patents, that's consistent: careful engineering on known problems rather than big architectural swings. The filing is worth understanding for anyone following AI-assisted graphics, but the practical ceiling depends entirely on threshold calibration that the patent doesn't specify.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
9 drawing sheets from US 2026/0301311 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in