Meta Patents a Way to Select Real-World Objects Using Your Eyes and Hand Shape
Pointing at something in augmented reality sounds simple, but getting a headset to know exactly which object you mean has always been the hard part. Meta's new patent combines where you're looking, how you're moving your hand, and what the cameras see to pin down your selection.
How Meta wants your eyes and hands to work together in AR
Today, selecting objects in augmented reality typically means pointing, tapping a button, or staring at a target long enough to trigger it. Each method works in limited situations, but none of them feel natural when the thing you want is ambiguous or surrounded by other objects.
Meta's patent describes a system that fuses three signals at once: where your eyes are looking (from the headset's built-in eye tracker), what the cameras on the headset can see, and the shape and size of a hand gesture picked up by a wrist-worn device, like a wristband with motion sensors. The system figures out which object, or group of objects, you're trying to select based on all three inputs together.
So if you glance at a cluster of items and make a small pinching gesture, the system might select one item. Make a wider, scooping gesture while looking at the same cluster, and it could select several. You get to communicate intent through natural movement rather than navigating menus.
… receiving, from the head-wearable device, gaze data comprising a user gaze at one or more of the one or more objects; receiving, from a wrist-wearable device worn by the user, hand gesture data indicating the user performing a selection hand gesture …
Translation: The system tracks where your eyes are looking and what your hand is doing using your headset and wristband.
How gaze, gesture size, and camera data merge into one selection
The patent describes a method that runs when a user performs what it calls a selection input. At that moment, the system simultaneously pulls in data from three sources:
- Image data from one or more cameras on the head-worn device, capturing everything in the user's field of view and identifying objects in it.
- Gaze data from the same headset, tracking where the user's eyes are pointing at that instant.
- Hand gesture data from a wrist-wearable device (think a smart wristband), which captures the shape and size of the gesture the user is making.
The key innovation is that the system uses both gesture size and gesture shape together with the eye-tracking data to resolve exactly which objects, or how many, the user is targeting. A small, tight gesture narrows the selection. A large, open gesture expands it. The system can therefore select a single object, a portion of a scene, or a freeform grouping.
This is different from current approaches where selection is usually binary: you either tap on one thing or draw a box. The patent's approach lets the gesture itself communicate the scope of the selection, not just the trigger. The result is presented back to the user as a visual indication that the chosen object or objects are selected.
… determining, based on the gaze data, the image data, the gesture size, and the gesture shape, at least a portion of the one or more objects, and presenting, to the user, an indication that the portion of the one or more objects is selected.
Translation: It figures out what you want to select by combining your eye direction with the size and shape of your hand movement.
What this means for hands-free AR control on Meta's devices
For anyone wearing AR glasses day-to-day, the selection problem is a constant friction point. Menus and tap interactions break the feeling that digital content lives naturally in the real world. A system that reads your gaze and the shape of your hand gesture together could make selecting content feel far closer to just reaching for something.
Meta has been filing around wrist-based and gaze-based input since at least 2021, which tracks with its acquisition of CTRL-labs and the direction of its Ray-Ban and Quest hardware. This patent pushes that work into more expressive territory, where the size and shape of a gesture carry meaning, not just the fact that one was made.
Meta's 72nd filing we've tracked in the AR glasses race since May adds to a run that includes body-movement window control and audio muting for privacy.
The shortest path from this patent to a shippable feature requires two pieces of hardware to exist simultaneously: a headset with reliable eye-tracking and cameras, and a wrist-worn sensor that can read gesture shape with enough precision to distinguish a small pinch from a wide scoop. Meta has both ingredients in various stages of development, so this is not a purely theoretical filing.
That said, the real challenge is the fusion step. Getting gaze, gesture, and vision data to agree on the right object in real time, across messy real-world scenes, is an engineering problem the patent describes but doesn't solve in detail. The claim is broad: it says the system determines the selected objects based on all four inputs, but leaves the actual algorithm open.
For now, this reads like Meta staking out the design space for a natural selection interface before the hardware matures enough to ship it. That is a reasonable strategy, and the direction is clearly the right one for AR to feel usable.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
18 drawing sheets from US 2026/0299683 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in