Nvidia Patents Software That Learns to Follow Your Eyes Without Scanning Real Faces
Training an AI to track where your eyes are looking normally requires thousands of real photos of real people's eyes. Nvidia's new patent describes a way to skip that entirely, generating artificial eye images so convincing that the AI can't tell the difference.
How Nvidia's synthetic eye images replace real photo shoots
Every time a VR headset tries to figure out where you're looking, a camera fires an infrared light at your eye and a small AI reads the reflection. That AI had to learn from countless photos of actual humans' eyes, which is slow and expensive to collect.
Nvidia's patent describes a system that builds those training images from scratch on a computer. It simulates the way infrared light bounces off an eye's surface, adjusts the shape of the face and eyelids around it, and produces photos that look like a real camera took them but never involved a real person.
The payoff is that eye-tracking AI can be trained faster, with more variety, and without requiring anyone to sit in a lab while cameras stare at them. That matters a lot for devices like VR headsets and smart glasses, where accurate gaze tracking is central to how the whole experience works.
… generate one or more synthetic images that depict the at least one eye in the one or more eye positions and the one or more reflections corresponding to infrared illumination with one or more modifications applied to geometric representations of at least one of the one or more eyes or a face comprising the one or more eyes in the one or more synthetic images.
Translation: It creates fake pictures of eyes and faces with altered shapes based on infrared tracking data.
How the system fakes infrared glints on a digital eye
The patent describes a processor-level pipeline with three main jobs.
First, it takes a description of where an eye is positioned and where it's looking. Second, it figures out what infrared light reflections (the small glints you'd see in a real IR camera image) would appear on the eye's surface at that position. Infrared cameras are the standard sensor inside eye-tracking headsets because visible light is too distracting.
Third, it generates a complete synthetic image of an eye in that position, including the surrounding face geometry, with those infrared glints baked in. The system can modify the shape of the eye, the eyelid, and the surrounding face to produce a wide range of variations, which is important because a good training dataset needs examples from many different people with different facial structures.
The neural networks in the system are trained to understand what eye-gaze information looks like in IR imagery, so the synthetic images they produce are calibrated to look realistic to another AI that will later be trained on them. This approach, sometimes called synthetic data generation, is used across AI research when real labeled data is hard or expensive to collect.
… calculating one or more activation values of one or more neural networks trained to infer eye gaze information based, at least in part, on eye position of one or more images of one or more faces indicated by an infrared light reflection from the one or more images.
Translation: An AI program figures out where someone is looking by analyzing infrared reflections off their face.
What this means for AR headsets and privacy in gaze tracking
For anyone who wears a VR headset or uses a laptop with built-in eye tracking, the accuracy of gaze detection is directly tied to how well the underlying AI was trained. A system trained on a narrow set of real faces tends to work poorly for people whose eyes don't match that sample, an ongoing and well-documented problem in computer vision. Synthetic data, generated with deliberate variation across eye shapes and face geometries, is one of the cleaner ways to close that gap without recruiting thousands of volunteers.
Nvidia's hardware already powers most of the consumer AR and VR market's graphics processing, so a strong software method for training gaze models fits naturally into their broader platform play. Eye tracking is also increasingly used for foveated rendering, a technique where the headset renders the spot you're looking at in high resolution while letting everything else drop in quality to save power. The newest Big Tech patents in the AR and gaze-tracking space show this kind of training-data infrastructure becoming as contested as the sensors themselves.
Nvidia's second filing we've tracked in the AR glasses race since July builds on their gaze-to-audio work with a new application.
VR headsets track where your eyes are looking. When that tracking fails mid-experience, it feels jarring and wrong.
This approach fixes the root cause. Instead of relying on limited real-world photos to train the software, it generates huge amounts of synthetic practice images using accurate simulated light. The software sees far more variety in eye shapes, skin tones, and lighting before it ever meets a real face. That means steadier tracking for more people, especially those whose eyes were poorly represented in earlier systems.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
8 drawing sheets from US 2026/0245346 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →