Apple Files Patent to Let You Invent Your Own Hand Gestures
Apple's filing describes a way to teach a camera-equipped device a hand gesture you made up, using as little as a single demonstration.
What Apple's custom hand-gesture patent does
Ever wished you could invent your own hand signal, like a pinch-and-drag or a little pair of scissors, and have your device simply know what it means? That is the idea here.
Apple's filing describes a camera-based system where you teach a device your own gestures. You pick a few gestures (the description mentions three to six), perform each one in front of a camera, and review the recordings on screen. The device then trains itself, which the filing says can take under a minute. After that, your gesture can trigger an output or an action.
The trick is that the device doesn't start from zero. It begins with a model already trained on a general set of gestures and adjusts it using your handful of examples. It also doesn't study raw video. It works from a stick-figure outline of your hand.
… extracting, by the electronic device, a skeleton-based feature representation from the image data; applying, by the electronic device, a trained machine learning model to the skeleton-based feature representation to generate a classification of the hand gesture as corresponding to a user-customized gesture class …
Translation: The device maps your hand bones into a digital skeleton and uses AI to match it to a custom gesture you invented.
How a stick-figure hand teaches the model
The claims cover a basic loop. A camera captures you performing a gesture, and the device converts the footage into a skeleton-based feature representation (a stick-figure map of your hand). The filing describes 21 landmark points per hand: three on each finger, five knuckles and one wrist. A trained model then labels the gesture as one of your custom gestures, and the device produces an output or runs an operation.
The model is the clever part. It starts as a graph transformer (an AI that studies how connected points, here finger joints, relate to each other) trained on a public gesture dataset. Meta-learning (practice at learning new tasks from very few examples) then prepares it so a single demonstration can be enough. To add variety, the training invents two-handed gestures by pairing up one-handed ones.
Around that core, the filing describes:
- A setup flow where you choose 3 to 6 gestures and record samples of each.
- A review screen that shows the skeleton so you can delete bad takes.
- Training on the device itself, which can finish in under a minute.
- A live preview, plus syncing the finished model across your devices.
By utilizing skeleton-based feature representations and incremental training of a preexisting model, the subject technology can improve gesture-recognition robustness across imaging conditions, reduce training data requirements for personalized gestures, decrease model-development overhead, and enable deployment of user-defined gesture controls.
Translation: This approach helps the device reliably recognize your custom movements without needing massive amounts of training data.
Why your own gestures could matter on Apple devices
If this ever reaches a product, you would pick the gestures yourself instead of memorizing a fixed set. Your watch, headset or laptop could learn a flick, a heart shape or a finger frame that means whatever you decide. The filing says custom gestures can be easier to remember, faster to use, and more inclusive for people with specific needs, such as physical disabilities.
Privacy gets a nod too. The filing says training that involves personal information can happen entirely on your device, and the model itself works from a hand outline rather than raw pixels. The listed devices include phones, watches, headsets, earphones and desktop computers, so the idea is not tied to one gadget.
Apple's 42nd filing we've tracked since May joins earlier work on bird chirps for attention and floating pinch controls in our eye and hand controls watch.
The camera this needs already exists on phones, laptops, and headsets, so no new hardware has to be invented. That alone puts this closer to a real feature than most patent filings.
The remaining work is software and setup design: someone has to build the screen where you record your gesture, decide what it triggers, and warn you when two gestures look too similar to tell apart. Training the system on your specific gesture, quickly, on the device itself, is ambitious on something small like a watch, so a phone or headset is the easier starting point.
The shortest path to a product is probably a settings option, perhaps in accessibility, where you record a gesture and assign it to a shortcut. Real rooms and real lighting stand between this filing and that screen, but the distance is shorter than usual.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Apple patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
14 drawing sheets from US 2026/0330877 A1 · click any drawing to enlarge
Show the other 6 sheets
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in