Microsoft
Microsoft · AI / ML · Filed May 7, 2025 · USPTO published Nov 12, 2026 · This week's release
Microsoft Files Patent for Teaching Robots to Grip Things by Watching Human Hands
Microsoft's patent filing describes a way to turn videos of people using their hands into reusable grip patterns, so a robot could handle a pickle jar after learning from a screwdriver.
- Publication number
- US 2026/0355960 A1(US20260355960A1)
- Applicant
- Microsoft Technology Licensing, LLC
- Filed · Published
- May 7, 2025 · Nov 12, 2026
- Inventors
- Benjamin Eliot LUNDELL, Harpreet Singh SAWHNEY, Dmitry Petrovich ANDREYCHUK, ADITYA PRAKASH
- Status when we published
- Examiner raised objections; the company can respond
- Document
- 20 claims
- US classification
- 700/245
- Examiner
- DO, TRUC M (Art Unit 3658)
- Parent application
- Continuation of 19/200,145 (filed May 6, 2025)(19200145)
What Microsoft's hand-copying robot patent does
Teaching a robot a chore works best when an expert films themselves doing that exact chore on that exact object. Want a robot to change the oil filter on your car? The ideal lesson is a video of someone changing that very filter. Nobody can film every jar, cup and lever in the world.
Microsoft's filing tries to get around that. The system watches videos of people using their hands, works out which parts of the hand touch the object and how those touch points move, and saves each move as a reusable pattern. Turning a screwdriver, for example, becomes a twist pattern.
Later, you show the system a photo of something new, like a pickle jar, and tell it what to do. It stretches the stored pattern to fit the new object and produces step-by-step hand instructions. Those could guide a robot hand, or a hologram showing a person what to do.
accessing a recording in which a gripping structure is performing an action against an object, wherein the recording includes time-series data illustrating performance of the action by the gripping structure over a period of time …
Translation: The system watches video footage of a hand or gripper interacting with an object over time.
How the system turns hand video into grip steps
The claims cover the learning step. The system builds a library the filing calls a latent codebook, which is a compact catalog of movement patterns, each stored as a string of numbers. It starts with a recording of a hand performing an action over time, such as rotating, pulling, pushing or sliding something.
From that recording, the system does three things:
- Builds a sequence of 3D poses of the hand, one for each moment in the video.
- Works out the contact points for each pose, meaning which parts of the hand are touching the object. These can be inferred when fingers are hidden from the camera, and they can be stored as binary masks (simple on/off flags) over a 3D hand mesh.
- Converts the full sequences into embeddings (compact numerical summaries) and adds them to the codebook.
The description goes further than the claims. Given a photo of a new object, a text or spoken instruction like 'pull the lever,' and a starting contact point, the system retrieves the closest stored pattern and reshapes it to fit the new object's size and shape. The output is a sequence of instructions with a new path of contact points, which can drive a robot hand or a hologram in a headset.
For training, the filing describes more than 2,000 first-person videos covering 24 action types and 120 kinds of objects.
The vectors in the latent codebook describe actions by mapping trajectories of contact points between gripping structures and objects during performance of actions.
Translation: It converts physical contact points into digital math that represents how a grip is executed.
Why robot training from human video matters
If it works, you could teach a robot a chore from the huge pile of videos of people already doing chores, instead of recording a costly set of robot demonstrations for every object. The filing says this approach reduces the cost of collecting robot data, and it notes that factory and assembly-line video could feed the models too.
The same output could power headset guidance: a trainee asks how a lever works, and a hologram shows the hand motion. The application continues an earlier one, and the examiner's first review rejected the claims as filed, which is routine at this stage and usually followed by amendments. Nothing in the filing says a product is on the way.
Microsoft's 31st filing in the AI vision work we've tracked since May adds to a run that includes one on locating phones and robots and one on reading any floor plan by building.
Source: US 2026/0355960 A1 at the USPTO (PDF). Full patent text and figures are in the official publication.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Microsoft patent news breaks.
Add Patentlyze on Google
One click.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings · 9 sheets
9 drawing sheets from US 2026/0355960 A1 · click any drawing to enlarge
Show the other 1 sheets
More from Microsoft
All 646 Microsoft patents →
More in AI vision
All 339 AI vision patents →
Nvidia
Nvidia Patents a Two-Stage AI Pipeline for Teaching Robots to Grab Objects
Nvidia
Nvidia Files Patent for Indoor Robot Maps That Update Instead of Starting Over
Nvidia
Nvidia Files Patent for Robot Vision That Redraws an Object Until It Matches the Camera
Nvidia
No comments yet