Microsoft Patents a Way to Find Your Location by Matching Photos to 3D Maps
GPS doesn't work indoors, inside tunnels, or in dense city blocks where satellites get blocked. Microsoft is patenting a method that figures out exactly where you are and which way you're facing by comparing a photo or a 3D scan of your surroundings to a pre-built map of that environment.
How Microsoft pins down location without GPS
Imagine you're wearing AR glasses inside a busy airport terminal. GPS is useless, so your device takes a quick snapshot of the hallway in front of you. Microsoft's patent describes a system that matches what the camera sees against a detailed 3D map of the building to figure out exactly where you are and which way you're facing.
The trick is that the system can work in both directions. You can give it a regular photo and it compares that to a 3D point cloud (a cloud of millions of dots that together form a digital version of a real space). Or you can flip it: give it a 3D scan and it compares that to stored 2D photos. Either way, the system finds matching features across both formats and uses those matches to pin down your position.
This kind of visual localization is especially useful in places where GPS fails, like warehouses, hospitals, underground stations, or dense city streets where tall buildings block satellite signals.
receiving a query comprising a three-dimensional (3D) point cloud depicting an environment of the entity, the 3D point cloud omitting visual imagery of the environment of the entity; …
Translation: The system starts by getting a 3D data map of your surroundings without needing any actual photos.
How 2D image descriptors match against 3D point clouds
The patent describes a localization system that finds the position and orientation (called a pose) of a device or person by cross-referencing two different representations of the same physical space.
On one side, you have a 2D image, a regular photograph. On the other, a 3D point cloud, a data structure made of millions of coordinate points that together describe the geometry of a real environment (think of it as a dense digital skeleton of a room or street). The system extracts descriptors from each, which are compact numerical fingerprints that summarize the visual or geometric character of a patch or region. It then correlates descriptors across the two formats, looking for pairs that are likely to represent the same real-world feature.
When a pair scores above a confidence threshold, it counts as a valid correspondence. The system accumulates enough of these correspondences to mathematically solve for where the device must be standing and which way it must be pointing, relative to the stored map.
The key claim in the filing covers the reverse direction too: instead of a photo as the query, the device submits a fresh 3D scan, and the stored map is built from pre-tagged 2D images that already have known positions baked in. This lets the matching run either way, 2D-to-3D or 3D-to-2D, using the same underlying descriptor-correlation approach.
… correlating the image descriptors with the point cloud descriptors to produce correspondences, wherein a correspondence is an image descriptor corresponding to a point cloud descriptor; …
Translation: It matches features from a 2D photo with features from a 3D map to figure out where you are.
What this means for AR headsets and indoor navigation
For anyone using an AR headset, a delivery robot, or an indoor navigation app, knowing your precise position and orientation is everything. A meter of error in a warehouse means the robot picks the wrong shelf. A degree of error in AR means virtual labels float off their real-world targets. Systems that depend purely on GPS can't solve this; GPS struggles or fails entirely indoors and in urban canyons.
Microsoft's interest in spatial computing and AR runs through products like HoloLens and mixed-reality platforms, where visual localization is a core requirement. A system that can match across 2D and 3D formats without needing GPS could make those experiences far more reliable in the environments where people actually use them.
Microsoft's 12th filing we've tracked since May in the AR glasses race builds on one sharpening display gratings and one pushing light outward.
The core design choice here is cross-modal matching: instead of keeping 2D and 3D data in separate pipelines, Microsoft is betting that a single descriptor-correlation step can bridge both formats. That's a real engineering commitment, because the two representations look nothing alike at the data level. A 3D point cloud has no color, no texture, just geometry; a photo has color and texture but no depth. Training a system to produce descriptors that are comparable across that gap requires significant effort and, almost certainly, large training datasets.
The tradeoff is brittleness at the edges. Descriptor matching is vulnerable to lighting changes, occlusion, and environments that look similar but aren't (long airport corridors, identical office floors). The patent's threshold mechanism helps filter weak matches, but it can't manufacture good correspondences when the environment simply doesn't give the system enough distinctive features to work with.
That said, the bidirectional design (query with a photo OR a 3D scan, match against whichever format the map stores) is a practical win. It means the same system works for different types of devices, cameras, and depth sensors, without requiring a separate localization pipeline for each. For a platform company building across headsets, phones, and robots, that kind of format flexibility matters more than it might look on paper.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
8 drawing sheets from US 2026/0278835 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →