Meta Patents an AI That Makes Virtual Lights Look Real in Mixed-Reality Scenes
When you place a virtual lamp in a mixed-reality scene, the surrounding room should look lit by it. That sounds obvious, but making it happen in real time, convincingly, is one of the hardest open problems in augmented reality.
What Meta's virtual lighting AI actually does to your view
Ever tried to put a sticker on a photo and have it look completely fake because the lighting was wrong? That same problem, scaled up to live video, is what makes mixed reality feel unconvincing.
Meta's patent describes an AI model trained to figure out how a virtual light source would actually change the look of a real scene. Feed it a frame of what your cameras see, depth information about how far away things are, and details about a virtual light (position, color, brightness), and it outputs a new version of that frame where the real-world objects appear to be lit by that virtual light. The model learns this from a training dataset, not from hand-coded rules.
The result is that a virtual candle on your table could cast warm shadows across your coffee mug, or a digital spotlight could make your wall look genuinely brighter. Your physical environment and the digital overlay would feel like they belong in the same room.
training, using a training dataset, a scene relighting model to incorporate an effect of a virtual light source on a mixed-reality scene, the training resulting in a trained scene relighting model …
Translation: AI is trained on data to figure out how digital lights should realistically brighten mixed reality environments.
How the relighting model blends depth, color, and light data
The system trains a scene relighting model (an AI neural network) on a dataset of paired examples: scenes with and without virtual light effects applied. Once trained, the model takes three inputs for any given video frame:
- Color data from the frame (what the cameras actually see)
- Depth data (a per-pixel map of how far away surfaces are, typically from depth sensors built into AR headsets)
- Virtual light source data (properties like position, color, and intensity of the digital light being added)
From those inputs it generates a modified version of the same frame, one where real-world surfaces appear to react to the virtual light, showing plausible shadows, highlights, and color shifts.
The depth data is key. Knowing how surfaces are oriented in 3D space lets the model reason about which parts of a room would face a light source and which would fall into shadow. Without it, a flat image-editing approach would smear light effects uniformly and look wrong.
The training approach means the system learns from examples rather than needing engineers to manually program every light-interaction rule. That makes it more flexible across different room geometries, surface materials, and lighting setups than classical rendering methods.
… generating, from first color data and first depth data of a first frame of a mixed-reality scene and data of a first virtual light source, using the trained scene relighting model, a second frame corresponding to the first frame …
Translation: The system uses color and depth details from a video frame to redraw it with accurate virtual lighting applied.
What this means for AR glasses and mixed-reality experiences
For anyone using mixed-reality glasses, bad lighting consistency is one of the biggest things that breaks the illusion. A virtual object that looks lit by an imaginary sun when your room lights are off, or a digital character standing in your kitchen with no shadow on the floor, pulls you out of the experience immediately. A model that solves this in real time, frame by frame, moves AR closer to feeling like the real world rather than a screen overlay.
Meta has invested heavily in AR and VR hardware through its Quest and Ray-Ban smart glasses lines, and this patent sits squarely in the technical stack those products need to improve. The latest Big Tech patents in the AR space show a broad push from multiple companies to close exactly these perceptual gaps between physical and digital environments, and Meta's relighting filing is a direct move in that direction.
Meta files its 51st application we've tracked in our AR glasses race watchlist since May, adding to work like one on dichroic mirrors and hand and eye gestures.
The frustration this solves is real. For years, fake objects dropped into AR scenes have looked wrong because the lighting on them did not match the room. That mismatch is not a minor flaw. It is one of the main ways your brain instantly spots something as fake.
Using AI to read the depth, color, and light in each frame is the right way to attack this. Simple rule-based systems failed because no two rooms are alike, and you cannot write rules for every possible space.
The big unanswered question is speed. Can this run fast enough on a headset so you never notice a delay? The patent does not say. The idea is sound. Making it work in real time is the hard part.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
7 drawing sheets from US 2026/0245297 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →