Sony Patents AI That Learns How Game Objects Look From Every Angle Before Streaming
Streaming a 3D game means sending an enormous amount of visual information just to make surfaces look correct from where you're standing. Sony's new patent tries to shrink that problem by shipping a tiny trained AI model instead of raw image data.
What Sony's server-side texture AI actually does
A player opens a cloud game on their TV or phone. The game world lives entirely on a remote server, which has to figure out how every surface should look from whatever angle the player is currently viewing it. Sending that as raw video every frame is a lot of data, and the picture can still look wrong when the player moves quickly.
Sony's patent describes a different approach: the server studies each object in the scene from many camera positions, uses that study to teach a small AI model what the surface should look like from any angle, then ships that compact model down to your device. Your device uses the model locally to generate the correct colors on the fly, without waiting for the server to render every frame from scratch.
The idea is that a small trained model is far cheaper to transmit than continuous high-resolution video, and once it arrives on your side, it can keep up with fast viewpoint changes in real time.
… generate a plurality of textures for learning representing color value distributions on a surface of an object, by rendering images of the object viewed from a plurality of viewpoints for learning generated according to the display viewpoint; …
Translation: The server takes screenshots of an in-game object from every possible angle based on where the player is currently looking.
How the server trains and ships a per-object color model
The system has two sides: a content server that does the heavy preparation work, and a client terminal (your TV, phone, or console) that handles the final display.
On the server side, a virtual camera is placed at many different positions and orientations around each object in the 3D scene. The server renders what the object looks like from each of those positions, producing a set of learning textures (essentially reference snapshots that capture how color and lighting vary with viewing angle). Those snapshots feed a machine-learning process that trains a compact model for each object. The model learns to take two inputs, UV coordinates (a flat 2D address on the object's surface) and a viewing direction, and output the correct color value.
That trained model, called a texture model, is then packaged with the geometry data (the 3D shape skeleton) and sent to the client. The client no longer needs the server to render every frame. Instead it runs the model locally to color each surface according to wherever the player's virtual camera happens to be pointing at that moment.
This is conceptually similar to Neural Radiance Fields (NeRF), a technique that encodes how a scene looks from any viewpoint into a neural network, though the patent focuses specifically on per-object texture models rather than a single scene-wide network.
… performs machine learning according to the learning textures to generate, for each object, a texture model configured to acquire a color value from UV coordinates and a direction, and transmits the texture models to a client terminal.
Translation: An AI uses these multi-angle images to build a compact visual model that it sends to the player's device.
What this means for cloud gaming and remote rendering
Cloud gaming has always had a bandwidth and latency ceiling. The server has to render a frame, compress it, ship it, and decompress it before you see anything, and that chain adds delay. If Sony's approach works in practice, the server ships a one-time AI model per object rather than an endless stream of rendered frames, which could mean lower data use and faster response to camera movement on your end.
The practical reach of this depends on how often game worlds reuse static objects (where the model stays valid) versus how often everything is changing dynamically. Sony Interactive Entertainment's steady stream of cloud-rendering filings suggests the company is treating this as a long-term platform bet, not a one-off experiment.
That makes this Sony's 15th filing in AI simulation we've tracked since May, a group that includes one on multi-angle game replay and one fixing corrupted motion data.
Claim 1 is written broadly enough to be interesting. It covers any server that renders an object from multiple learning viewpoints, runs machine learning on those renders to produce a viewpoint-aware color model, and ships that model to a client. There is no restriction to a specific neural-network architecture, a specific game engine, or even a specific type of object. That breadth means, if granted, the claim could theoretically reach a wide range of cloud-rendering pipelines that follow this general training-then-transmit structure.
The narrowing details live in the dependent claims, which are not shown here. The independent claim's language about 'a plurality of viewpoints for learning generated according to the display viewpoint' is the most interesting phrase: it implies the server tailors its training set to where the player is actually looking, rather than pre-training on every possible angle blindly. That adaptive angle could be the real novelty the examiner focuses on.
In practice, the claim would face scrutiny against prior NeRF-based streaming research, which covers much of the same ground. Whether Sony's framing of per-object texture models as a distinct invention over scene-level neural rendering survives examination is the real question this filing has to answer.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
10 drawing sheets from US 2026/0268586 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →