Sony Patents a Way to Teach AI the Full 3D Layout of a Game World
Sony has filed a patent for a system that trains an AI model on a running game, teaching it the full three-dimensional shape of the game world so it can generate views the game engine itself never explicitly drew.
How Sony's system learns a game world from the inside
Every time a game renders a frame on your screen, it only shows you one specific camera angle. The scene behind you, above you, or just off to the side exists in the game's logic, but your console never actually draws those pixels unless the camera swings that way.
Sony's patent describes a system that runs alongside a game and uses extra viewpoints to train an AI on what the world looks like from many angles at once. The AI builds its own internal map of the three-dimensional space, not just the frames you see. Once that map is built, it can generate images from angles the original game engine never rendered directly.
The system also includes a safeguard: game developers can set viewpoint restrictions, so the AI is only allowed to learn from camera positions the developer approves. That matters for games where certain angles would break the experience, reveal hidden geometry, or expose content players aren't supposed to see.
… generate, at a predetermined rate, a frame of a display image representing a three-dimensional display world where a situation changes in accordance with a user operation; and generate a training image different from the display image and representing the display world; …
Translation: The system continuously renders both the normal game view and a separate training image for the AI.
How viewpoint limits and training images work together
The patent describes an image processing device that runs in parallel with a game application. At its core, it does three things:
- Captures training images: While the game runs normally, the system generates additional frames from viewpoints the player never sees. These aren't shown to the user; they exist solely to teach the AI.
- Builds a 3D scene model: A component called the 3D scene information generation unit applies machine learning to those training images, producing a data structure that encodes the three-dimensional shape, appearance, and depth of the game world. This is conceptually similar to techniques like Neural Radiance Fields (NeRF), which reconstruct a 3D space from a set of 2D images.
- Renders novel viewpoints: An arbitrary viewpoint image generation unit uses that 3D model to produce images from camera positions that were never explicitly rendered by the game engine.
Critically, the whole process is gated by viewpoint restriction information tied to the application program. The developer defines which camera angles are valid for training and for output, so the AI cannot wander outside those approved boundaries. The training happens at a set rate alongside normal gameplay, meaning the system learns the world incrementally as the game runs.
An additional viewpoint setting unit of a content server acquires viewpoint restriction information from an application execution unit, and sets viewpoints for generating training images within a restriction range.
Translation: A server restricts where the AI can look to keep its training within the game boundaries.
What this could mean for how games look and stream
For players, the practical payoff would show up in situations where the game camera needs to move fluidly to angles that weren't pre-baked by the developer, like a cinematic replay, a spectator mode, or a dynamic virtual camera in a sports or action title. Instead of the engine struggling to render geometry it wasn't optimized for, the AI fills in those views directly.
The viewpoint restriction system is the detail that makes this workable in practice. Sony Interactive Entertainment's consistent filing around real-time rendering suggests the company is thinking about how AI-generated imagery fits inside the existing rules of game development, not as a replacement for the engine but as a layer that extends what it can show without breaking what the developer intended.
This is the 16th Sony filing we've tracked in our AI simulation coverage since May, building on teaching AI object angles and any-angle game replay.
If you've ever watched a replay in a sports game and seen the camera clip through a wall or snap to a broken angle, this patent is aimed directly at that moment. The system teaches the AI to understand a game world only from positions where the camera is actually allowed to go, so it can never generate a view it was never meant to show you.
The other change you'd benefit from is invisible: the AI learns the world during normal play, without developers building anything extra by hand. That means free-roaming cameras and flexible replays become cheaper to include, which makes them more likely to ship in games that would have skipped the feature entirely.
Whether you'd ever notice it working depends on whether the result looks as clean as ordinary graphics at full speed. The architecture here is thoughtful, but the real test is a finished product in your hands, and that hasn't happened yet.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
20 drawing sheets from US 2026/0268583 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →