Google
Google · AR / VR · Filed Jul 20, 2026 · USPTO published Nov 12, 2026 · This week's release
A New Google Patent Uses Your Hand to Work Out What You Mean by 'This'
Google's patent application teaches a voice assistant to figure out what 'this' means by looking at where your hand is. Ask 'What is this?' while reaching toward a bottle, and it answers out loud.
- Publication number
- US 2026/0360015 A1(US20260360015A1)
- Applicant
- Google LLC
- Filed · Published
- Jul 20, 2026 · Nov 12, 2026
- Inventors
- Mr. Daniel V. Klein, Mr. Ramprasad Sedouram
- Status when we published
- Waiting for an examiner
- Document
- 20 claims
- US classification
- 382/162
- Parent application
- Continuation of 18/364,579 (filed Aug 3, 2023)(18364579)
How Google's assistant reads your pointing hand
Ever stood in front of two identical bottles and couldn't tell the shampoo from the conditioner? A friend standing next to you would just glance at where you're pointing when you ask, 'What is this?'
This Google patent application teaches a voice assistant to do the same. Your phone or smart glasses watch the scene through the camera, hear your question, and notice which object your hand is closest to. That tells the assistant what 'this' means, even when you never name the thing.
Then it works out the answer and says it out loud, such as which bottle is shampoo. The filing's examples also include checking whether a shirt and pants match in color, and warning a parent about a child's medicine dose.
… extracting visual context data from the image data, wherein the visual context data comprises an identification of a physical hand of the user and a proximity of the physical hand to one or more physical objects within the area of interest; …
Translation: The system looks at where your hand is pointing or touching to figure out what you are talking about.
How the assistant picks the object you mean
The claims describe a voice assistant that combines three inputs: camera footage of the area you're looking at, audio of a spoken question that refers to an object without naming it, and what the software can see of your hand.
The software finds your hand in the image and measures how close it is to each object in view. It uses that proximity to disambiguate the question (settle which of several possible things you meant), then works out an answer and speaks it using synthesized audio. The claims cover phones, headsets and smart glasses as the device.
The description fills in the rest:
- A wake phrase such as 'Ok computer', a tap on the glasses, or a button press starts speech recognition (software that turns speech into text).
- A visual feature recognizer reads label text and colors, so it can tell a blue bottle from a white one even when the label has worn off.
- If the assistant is unsure what you mean, it asks a follow-up question.
The title and abstract focus on a color-matching example, where the assistant compares the colors of a tie, shirt and pants and says whether they go together. Answers can also show up as on-screen overlays or as a buzz from the device.
… processing the query and the image data to: identify, for each particular object of the plurality of objects referred to by the query, one or more corresponding colors of the particular object; and determine, based on the corresponding colors, the color matching insight associated with the plurality objects for the digital assistant to discern.
Translation: It analyzes the colors of the items near you to give you advice on how well they match.
Why vague questions to glasses could get easier
Voice assistants tend to stumble on the way people actually talk. We say 'this' and 'that' and expect the other person to follow our eyes and hands. A system that handles that could make smart glasses or a phone camera far easier to use hands-free in the kitchen, the bathroom or a clothing store.
The filing's examples show where it might help most: reading a label in a language you can't read, checking whether a food is safe for a child with an allergy, or finding the right spices for a recipe. It also describes simple overlays, like a green halo or a red X, so a child who can't read could still tell which tube is toothpaste.
Google's 1056th filing in our Google coverage since May adds to a run of image-fixing ideas that includes rejected image editing and group photo face borrowing.
Source: US 2026/0360015 A1 at the USPTO (PDF). Full patent text and figures are in the official publication.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Google patent news breaks.
Add Patentlyze on Google
One click.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings · 9 sheets
9 drawing sheets from US 2026/0360015 A1 · click any drawing to enlarge
Show the other 1 sheets
More from Google
All 1,079 Google patents →
More in AR / VR
All 690 AR / VR patents →
Samsung
Samsung Files Patent to Keep Headset Hand Gestures Accurate While Your Head Moves
Apple
Apple Files Patent for Headset Controls That Treat Each Hand Separately
Apple
New Patent Lets Apple Devices Hand Off Control Based on Room Position
Meta
No comments yet