Apple Patents an AI System That Fills in the Images Your Vision Pro Hasn't Rendered Yet
Apple is working on a way for its AR headsets to fake frames they haven't actually rendered, using AI to predict what you'll see next before your eyes even get there.
How Apple's headset cuts corners your eyes won't notice
AR and VR headsets today have to render every single frame of video at full quality, which takes enormous computing power and drains batteries fast. Apple wants to change that by letting an AI fill in the gaps.
The idea works in two parts. First, for real-world video coming through the headset's cameras, a trained AI predicts what the next frame will look like based on where your body is and the shape of the room around you. Instead of the camera capturing every frame, the AI guesses some of them. Second, for the computer-generated graphics layered on top, the headset tracks exactly where your eyes are pointing and renders that specific spot in full detail. The area around it, which your eyes aren't focused on and you'd barely notice anyway, gets rendered at lower quality.
The result is that the headset does less work overall, while the experience looks the same to you. Your eyes only ever see full sharpness where they're actually looking.
… predicting a future gaze position based on the current body position parameters; rendering, at a first resolution, a gaze region of a frame in accordance with the future gaze position; and predicting a peripheral region of the frame outside the gaze region …
Translation: The headset guesses where you will look next and renders that spot in high detail while filling in the edges.
How the system splits each frame by where your eyes point
The patent covers two related problems in AR/VR displays and proposes a single architecture to address both.
Temporal supersampling for real-world frames: When cameras on a wearable device capture the physical environment, they do so at a certain frame rate. Rather than capturing every frame, the system passes existing frames, the wearer's body position data, and geometry information about the scene (a depth map of the room, essentially) into a deep learning model. That model predicts the intermediate frames that weren't captured, filling in the timeline at a higher effective frame rate without extra camera work.
Foveated rendering for virtual content: The system tracks gaze position, meaning exactly where the wearer's eyes are pointed, and predicts where the gaze will be in the near future. The region the eyes will focus on gets rendered at full resolution. Everything outside that central region, what vision scientists call the peripheral field, gets predicted at a lower resolution. The two regions are stitched together into a complete frame and sent to the display.
- Body position parameters feed both pipelines
- A trained neural network handles frame prediction for real-world content
- Gaze prediction drives resolution allocation for virtual content
- Final composited frames drive the headset display directly
The first claim is specifically about the gaze-and-peripheral split for virtual rendering, making that the core legal boundary of the filing.
A peripheral region is predicted for the frame at a second resolution, and the combined regions form a frame that is used to drive a display.
Translation: The predicted side areas and the focused center are stitched together to power the screen.
What this means for the next generation of Apple Vision Pro
Rendering quality is one of the biggest constraints holding back wearable AR/VR. The more frames a headset has to draw, and the higher the resolution, the more heat it generates, the faster it drains the battery, and the heavier the processor needs to be. Apple's Vision Pro already runs warm under heavy use. A system that intelligently skips work the human eye won't detect could let a future headset run longer, run cooler, or pack in sharper graphics without getting bigger.
Apple's interest in on-device AI for display rendering shows up across several recent filings, and this one sits squarely in that area. For users, the practical promise is simple: better-looking AR glasses that don't need to be plugged in every hour.
Apple's 78th filing we've tracked since May in our AR glasses work builds on applications like one sharpening eye-aimed displays and one cleaning up grainy see-through video.
Claim 1 locks down a specific three-step chain: predict where the user's eyes will land, render that spot in full detail, and fill in the surrounding edges at lower quality. All three steps together, in that order, define what the patent actually covers. That tight sequence makes this a narrow filing, not a sweeping lock on AI-assisted video in headsets.
That narrowness matters in a practical way. The patent only blocks the particular combination of gaze prediction followed by a two-tier image split based on that prediction. Anyone skipping the gaze-prediction step, or applying quality tiers differently, falls outside what claim 1 would protect if granted.
For the person wearing the headset, the technique means the device spends its computing power exactly where your eye is already heading, making sharp images less expensive to produce. That shows up as longer battery life, crisper visuals, or both, depending on how Apple ships it.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
8 drawing sheets from US 2026/0301118 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in