Sony Files Patent to Shrink Game Streams by Cutting Holes in Frames
Sony's patent deliberately punches holes in video game frames to save bandwidth, then leaves a few tiny hint pixels so a second AI can fill them back in without making things up.
What Sony's hole-punching image trick does
Ever had a cloud game stream turn laggy because your internet couldn't keep up? Every frame is a lot of data to push down the line, and less data means less waiting. Sony's patent tackles that by sending less picture in the first place.
Here's the trick. An AI looks at each frame and erases chunks it thinks can be rebuilt later, like big patches of sky or ground. In the empty patches it leaves a few anchor pixels, tiny reminders of what was really there.
Your device then runs a second AI that fills in the blanks, using those anchor pixels as hints. Without them, AI refilling tends to invent details that were never in the original image. With them, the filing says, the system can erase bigger areas and still get a faithful picture.
… removing a plurality of image portions from the image, using a machine learning inference process implemented by the first data model, to thereby insert a plurality of holes in the image …
Translation: AI cuts out chunks of the video frame to save transmission space.
How the AI picks what to erase and what to keep
The filing describes two matched neural networks (software that learns patterns from examples). A decimation network runs on the sender's side and removes parts of an image. A reconstruction network runs on the receiving device and fills the gaps back in. The claims cover both halves, plus a computing device and a system, and one claim names an extended reality (VR or mixed reality) device as the receiver.
The first network takes a full frame plus an importance map, which gives every pixel a score from 0 to 1 for how much it matters. Pixels scoring below a threshold (0.4 in the filing's example) are zeroed out in a mask and erased, while the rest stay. The network also receives a grid of possible anchor points and decides which ones are needed, then puts those pixels back into the holes.
The network can also take a target image quality or compression level as input. Lower quality or heavier compression means more or larger holes. In one embodiment, anchors only go into holes above a certain size.
Both networks are trained together on video game frames. A scoring step (a loss function) has two parts: how good the rebuilt image looks, and how compact the stripped image is. The remaining data then goes through a standard codec (a compressor) before it is sent.
What this means for cloud game streaming
The filing's own example is a cloud game streaming system, where the picture is rendered on a server and sent to you. Sending less data per frame could mean less waiting before each frame reaches your screen, which is what you feel as lag. The filing also names VR and mixed reality devices as possible receivers.
The weak spot of this kind of approach is AI inventing details that were never there. The anchor pixels are meant to keep the rebuild honest, which is what lets the system erase larger areas. The filing gives no figures for how much data it saves, so the real-world payoff is unproven.
This is the 12th Sony filing we've tracked in AI image & video since May, a topic that already includes one on post-shot camera settings and one on consistent video frame editing.
If this works, you would notice it as a game stream that stays sharp and responsive on a shaky connection, with no setting to change. The saving happens before the picture ever leaves the server, so you never see the machinery.
The risk sits in the rebuild. Any AI that fills in missing pixels can invent details, and in a fast shooter a made-up smudge near an enemy would be annoying. The anchor pixels are the filing's answer to that, and it is a sensible one.
The filing gives no numbers on how much data is saved, so the benefit stays a promise until someone tests it. Even so, spending a little extra computing on both ends to send less over the wire is a bargain most people who stream games would take.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Sony patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
6 drawing sheets from US 2026/0340791 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in