Nvidia Files Patent for Judging an Object's Shape and Lighting From Two Photos
A photo of your sneaker dropped into a video game rarely looks right, because a picture says nothing about how glossy the shoe is. Nvidia's patent application tries to read that from two photos of the object, taken under different lighting.
What Nvidia's two-photo object scan does
Today, if you photograph a sneaker and drop it into a video game, it usually looks pasted on. A photo records color, but it says nothing about how glossy or rough the surface is, so the sneaker never catches the game's light the way everything around it does.
Nvidia's filing describes a fix that needs only a phone camera. You take two shots of the same object, one with the flash and one without. Software trained on many examples (a neural network) studies the pair and works out the object's shape and the lighting in the room.
Then it checks its own work. It redraws the object from those guesses, compares the result with a real photo, and corrects what it got wrong. The goal is a digital copy that reacts to light like the real thing.
receive a first image of an object captured under first illumination conditions, and a second image of the object captured under second illumination conditions different than the first illumination conditions …
Translation: The system takes in two photos of the same item taken under different lighting.
How the networks guess, redraw and correct
The main claim covers processors with circuitry that takes two images of one object, shot under different lighting, such as with and without a flash. Neural networks (software that learns patterns from examples) make a first estimate of the object's shape and of the lighting. They then produce a rendered image, meaning a picture drawn by computer from those estimates, and compare it against the second photo to make a better estimate.
The description lays out a four-network chain, where each step is guided by the one before it:
- Shape: estimates depth (how far away each point is) and surface direction (which way each patch faces).
- Lighting: guesses the light in the scene, using the shape as a guide.
- Material: estimates color, shininess and roughness at every pixel.
- Refinement: redraws the flash photo, compares it with the real one, and fixes all the estimates together.
The filing says each network is small enough for phone hardware. The full chain takes about 700 milliseconds on a 256 by 256 image using an Nvidia 1080 Ti graphics card. Training used made-up scenes built from spheres, cones, cylinders and rings covered in 11,250 material maps, with the no-flash view shifted slightly to mimic a shaky hand.
… shape and visual properties of an object can be determined using differently illuminated images fed to a pipeline of neural networks …
Translation: A series of AI models figures out the physical form and lighting details from the photo pair.
What this means for scanning real objects
If this works as described, scanning a real object for a game or an AR scene could take two quick phone shots instead of a studio setup. The result would be a digital copy that reacts to virtual lighting, so it matches the objects around it. The filing names AR, VR and gaming as the motivation, along with growing demand to put physical objects into virtual worlds.
The filing is a continuation of an application first filed in February 2020, so the underlying research is not new. It also describes a handheld, burst-capture setup with slight misalignment between shots, which suits phones. Nothing in the text names a product.
Nvidia's 87th filing in the AI vision filings we've tracked since May adds to a run that includes one on mood-shifting characters and one on sports camera direction.
Read the main claim closely and it is wider than the description suggests. It asks for two images under different lighting, a first guess at shape and lighting, a rendered third image, and a second guess based on comparing that render to the second image. Flash is never mentioned, and neither are phones, gloss or the four-network chain.
As written, the claim covers any processor running that guess, redraw, compare and correct loop on a differently lit photo pair, whatever the hardware or the lighting trick. The flash pair and the small phone-sized networks sit in the description, and would only shape the claim if narrower claims were added.
Whether a claim this open survives review is an open question. It does show Nvidia is aiming at the whole idea of the loop.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Nvidia patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
49 drawing sheets from US 2026/0350770 A1 · click any drawing to enlarge
Show the other 41 sheets
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in