Sony Patents an AI Video Upscaler That Tracks Motion Frame by Frame
Sony is patenting a way to make low-resolution video look sharper by teaching an AI to remember where every pixel was in previous frames, then deliberately nudging those memories with a tiny dose of controlled randomness.
What Sony's frame-stacking upscaler actually does
Imagine you're watching a sports replay and the camera pans quickly across the field. Your TV is trying to fill in detail that wasn't in the original footage, but fast motion makes that hard because the picture from one moment looks very different from the next.
Sony's patent describes an AI system that solves this by keeping a running memory of every frame it has already seen. When the next frame arrives, the system first figures out how much each part of the image has moved, then uses that movement data to line up its memory correctly before the AI tries to sharpen the new frame.
The clever twist is a dash of controlled randomness: before passing that memory to the AI, the system shifts some pixel values to slightly random positions. That sounds counterintuitive, but it helps the AI avoid over-committing to artifacts from earlier frames, producing a cleaner, sharper result across the whole video.
How motion data and random offsets feed the AI model
The system processes a video sequence one frame at a time. For each new frame, it maintains what the patent calls cumulative feature information, essentially a compact summary of everything the AI has learned about the image so far, stored at the same pixel count as the original low-resolution input.
Before that summary is handed to the AI for the next frame, the system does two things:
- It measures movement information (optical flow, meaning the direction and speed each pixel traveled between the previous frame and the current one).
- It applies a pseudorandom offset, moving some pixel values to slightly shifted positions based on that flow data. Pseudorandom means the shifts follow a seeded mathematical sequence rather than true chaos, so the process is repeatable during training.
This adjusted summary, called auxiliary information, feeds into a machine learning model alongside the current raw frame. The model has two distinct stages: one that updates the running memory, and one that uses that memory to produce an estimated frame with a higher pixel count than the original input.
The training process pairs low-resolution frames with high-resolution targets so the model learns, over many examples, to reconstruct fine detail rather than just blurring things together.
What this means for Sony TVs and game console upscaling
Video upscaling is already built into Sony's Bravia TVs and PlayStation consoles via their proprietary processors. A technique that carries temporal memory across frames, corrected for motion, addresses one of the most common failure modes in AI upscaling: smearing or ghosting around fast-moving objects. If this approach ships in consumer hardware, it would most directly affect sports broadcasts, action films, and gaming content where motion is constant.
For the games industry specifically, upscaling that stays sharp through motion is a direct competitor to Nvidia's DLSS and AMD's FSR, both of which use similar frame-history strategies. Sony already has its own upscaler (PlayStation Spectral Super Resolution) and this patent could represent the next iteration of that work.
This is a technically specific and well-scoped patent, not a vague land-grab. The pseudorandom pixel-shifting trick is an interesting detail that distinguishes it from generic temporal upscaling filings. Whether it actually outperforms current solutions is impossible to say from a patent alone, but Sony clearly has the hardware ecosystem, Bravia TVs, PlayStation, and professional monitors, to ship this if it proves out.
Which company should we read for you?
We track 17 companies here. Pro is the same weekly breakdown for any company you choose, delivered privately. Type a name and we'll scope it and send you a quote.
Get one Big Tech patent every Sunday
Plain English, intelligent commentary, no hype. Free.
Editorial commentary on a publicly published patent application. Not legal advice.