Sony Files Patent to Cut Lag and Glitches in Cloud-Streamed 3D Games
Instead of streaming finished video frames, Sony's patent has a server teach a small AI model what a 3D scene looks like, then ships that model to your device to draw the picture locally.
What Sony's scene-sending cloud setup actually does
You're playing a game streamed from a distant server and you whip your head around in a VR headset. The picture lags behind, or a chunk of it vanishes because some data got lost on the way. That delay and those glitches are the problem Sony is going after.
In Sony's approach, the server doesn't send finished video. It renders the game world from lots of nearby angles, then trains several small AI models on those pictures. Quick-to-train, low-detail models go out first, so your device can draw a rough view almost right away. Sharper models follow and the image gradually clears up.
Because your device draws each frame from your latest head position, the view keeps up with your movement. The server also splits and shuffles the data it sends, so a few lost pieces don't wipe out the picture.
generate, as training images, images representing each scene of a three-dimensional display world viewed from a plurality of viewpoints, wherein a state of the three-dimensional display world is configured to change according to an operation; …
Translation: The server creates training pictures of every game scene from multiple angles that change based on player actions.
How Sony trains, splits and shuffles the scene data
The claims cover a content server and a client device (a game console, headset or phone). The server renders each scene of a 3D world from many viewpoints and uses those images as training data. It trains neural networks (AI models that learn patterns from examples) so each one captures the 3D layout and color of the scene. The description names a technique called NeRF (neural radiance fields) as one example, though it says the method isn't limited to that.
The server trains several versions of the same scene at once, and the key claim is that it transmits each one in the order it finishes training. Lower-detail versions finish first and go out first. The versions can also differ in how much of the world they cover, for example a small patch around where the player is looking (picked using eye tracking) versus the whole scene.
The filing also describes robustness against lost data:
- Each network is split into several smaller networks by dropping different random nodes (the individual units inside an AI model).
- The pieces are packetized, and their order is randomly shuffled so a burst of lost packets doesn't take out one whole scene.
- If every piece arrives, the client rebuilds the original network. If some are missing, it renders from what it has.
The server places its training viewpoints inside a sphere around your current view, sized by the fastest your viewpoint is expected to move.
… divides each of the plurality of neural networks into packets, randomizes an order of packets, and transmits the packets to a client terminal.
Translation: It chops up the AI data into mixed up chunks and sends them over to the user's device.
Why this matters for cloud gaming and VR headsets
If this works as described, your cloud-streamed game or VR session could show something usable almost immediately after the world changes, rather than waiting for a full-quality frame. Turning your head would be answered by your own device instead of waiting on a round trip to the server. A weak connection might mean a slightly blurrier view instead of a frozen or broken one.
It also fits where Sony's business sits: PlayStation, headsets and streaming. The filing names games, head-mounted displays and portable terminals. It is a patent application and describes no product, so treat it as a look at how Sony thinks about streaming rather than a feature coming to your console.
Sony's 18th filing we've tracked since July in the GPU rendering race follows earlier applications on sharpening streams and predicting frames ahead.
The cost of this design is that the server does far more work. Instead of rendering one picture per player, it has to render many extra viewpoints and train several AI models on them, over and over as the game world changes. That is a heavy bill for the data center, and it multiplies with every player in a match.
The design also bets that your device can turn those models into pictures fast enough, and that the guessed viewpoints actually cover where you end up looking. If you move faster than the server predicted, the view could fall apart.
Still, the trade reads as sensible for headsets, where lag makes people feel sick and a briefly blurry picture is easy to forgive. Whether the extra server cost pays off is a question the filing doesn't answer.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Sony patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
9 drawing sheets from US 2026/0330749 A1 · click any drawing to enlarge
Show the other 1 sheets
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in