Qualcomm Patents a Way for AR Glasses to Answer Questions About Any Object You Point At
What if you could tap on any unfamiliar object in an AR headset and just ask it a question? Qualcomm has filed a patent for exactly that, using a language model to generate context about real-world objects the device has never seen before.
What Qualcomm's point-and-ask AR system actually does
Imagine you're wearing AR glasses in a factory, and you spot a piece of machinery you've never seen before. You tap on it and ask, "What does this do?" The glasses don't have a pre-loaded database of every machine on earth, but Qualcomm's patented system figures it out anyway.
Here's the trick: when you tap on a spot on that object, the glasses know exactly where your finger pointed in the room. The system then converts that location into the object's own internal coordinate system, so it always knows which part of the object you touched, even if the object moves or rotates. That precise location, paired with your question, gets handed off to an AI language model.
The language model then generates a relevant answer based on the domain you're working in, whether that's manufacturing, medicine, or retail. It's less about recognizing the object from a library and more about letting AI reason through an unfamiliar situation using the spatial context you gave it.
… transform, based on a pose between the world coordinate system and the local object coordinate system, the first set of coordinates that represent the second user input in the world coordinate system to a third set of coordinates in the local object coordinate system; …
Translation: The system maps where you pointed in the real world to the exact spot on the 3D object model.
How the coordinate translation ties your tap to the object
The patent describes a processing system, likely inside an XR headset or a chip powering one, that handles three kinds of input at once:
- A verbal or text query from the user, tied to a specific domain (like "maintenance" or "retail")
- A spatial tap or pointer that identifies a location on a real-world object
- Object state information, which includes the object's own internal coordinate grid (called a local object coordinate system) and its orientation in space
The core engineering challenge here is a coordinate translation step. When you tap a spot in the real world, that location is recorded in world coordinates, meaning a fixed grid tied to your environment. But objects move. If someone rotates a box or a robot arm shifts, world coordinates alone won't tell you which part of the object you touched.
The system solves this by computing the pose (the position and rotation relationship) between the world coordinate system and the object's own local coordinate system. It transforms your tap's world coordinates into the object's local coordinates. That locked-in local position is then paired with your query and passed to a language model (an AI similar in concept to what powers chatbots), which generates a contextually relevant response.
The domain parameter matters here. The same tap on the same object could produce a very different answer depending on whether you're asking as a technician, a shopper, or a doctor. The system routes the language model's reasoning through that domain lens.
… generating, with a language model, an output based on the first user input, the first association, and context associated with the domain.
Translation: An AI language model uses your question and the specific pointed spot to generate an answer.
What this means for AR headsets that meet the real world
AR headsets have always struggled with the gap between the digital and physical worlds. Most current systems work by pre-labeling known objects, which means anything outside the database gets ignored. Qualcomm's approach flips that assumption: instead of requiring prior knowledge, it uses spatial precision plus AI reasoning to handle anything a user points at.
For industries like manufacturing, logistics, or field service, this could mean workers get instant, context-aware guidance on unfamiliar equipment without a pre-built knowledge base for every machine. For consumers, it inches AR glasses closer to a genuinely useful tool rather than a novelty. the pattern in Qualcomm's XR and on-device AI filings suggests the company is betting that the hardware intelligence layer, not just the display, is where AR wins or loses.
Qualcomm's 51st filing we've tracked since July in the AR glasses race adds to a pattern that includes focused hologram delivery and GPS-locked spatial audio.
The real cost of this design is the constant work it takes to know exactly where every object is pointing in space at any given moment. That calculation has to run continuously, on top of everything else a headset is already doing, which pushes against the battery and heat limits these devices already struggle with.
The precision is the whole point, and it does make the system more useful, especially in settings like factories where touching the wrong part of a machine and getting the wrong answer could matter. But the tradeoff reads as fragile the moment conditions get messy: poor lighting, a hand briefly covering part of the object, or anything moving faster than the sensors can track.
The patent describes the mechanism clearly, but says little about how gracefully it fails when those conditions hit. That gap is where the design will either prove itself or not.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
11 drawing sheets from US 2026/0301329 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in