Microsoft
Microsoft · AI / ML · Filed May 6, 2025 · USPTO published Nov 12, 2026 · This week's release
Microsoft Files Patent for Teaching Robot Hands to Handle Objects They've Never Seen
Teach a robot hand to turn a screwdriver, and this Microsoft patent application tries to let it work out a pickle jar lid without a new lesson.
- Publication number
- US 2026/0355959 A1(US20260355959A1)
- Applicant
- Microsoft Technology Licensing, LLC
- Filed · Published
- May 6, 2025 · Nov 12, 2026
- Inventors
- Benjamin Eliot LUNDELL, Harpreet Singh SAWHNEY, Dmitry Petrovich ANDREYCHUK, . ADITYA PRAKASH
- Status when we published
- Waiting for an examiner
- Document
- 20 claims
- US classification
- 382/155
- Examiner
- CROCKETT, JOSHUA BRIGHAM (Art Unit 2661)
What Microsoft's hand-motion library does
A robot that has never met a pickle jar usually has no instructions for opening one, and the example data needed to teach it is hard to come by. People skip that problem. If you can turn a screwdriver, you can figure out a jar lid. Microsoft's new patent application tries to give machines the same shortcut.
The system studies recordings of human hands doing everyday things like twisting, pulling and pushing. It boils each motion down to a compact pattern: which parts of the hand touch the object, and how those touch points travel. Those patterns go into a library of motions.
When a robot faces an unfamiliar object, it takes a photo and gets a short instruction such as 'twist the lid.' The system then stretches a stored pattern to fit the new object's size and shape, and produces step-by-step hand movements for the job.
accessing a latent codebook comprising a vector, wherein the vector describes a specific action that is performed by a first gripping structure against a first object, wherein the vector describes the specific action by mapping a trajectory of contact points between the first gripping structure and the first object during performance of the specific action …
Translation: The system looks up a stored mathematical description of how a robotic hand touches and moves an original item.
How a screwdriver twist becomes a jar-lid twist
The filing describes two stages. First, the system builds a latent codebook, which is a compact library of numeric codes where each code stands for a type of hand motion. It does this by watching video of a hand performing an action, building a 3D model of the hand in every frame, and marking where the hand touches the object. Those touch points are stored as on/off masks over the hand model. The full sequence is called an interaction trajectory.
Second comes the part the claims focus on. The system takes three things:
- A code from the library for one action, recorded on a first object;
- An image of a different, unmapped object;
- An instruction for the action, which can be text or audio.
It then localizes the code, meaning it adjusts the dimensions recorded for the first object to match the second object in the image. One described method shifts a contact point to a new position and assigns it to a different finger of the second hand, to account for a difference in size.
The output is a sequence of instructions with a new trajectory of contact points. It can include synthesized hand poses, 3D meshes or masks, and can go to a robot with a robotic hand or drive a hologram in an extended reality scene.
In the described setup, the library comes from a type of neural network (a model that compresses data into a fixed set of codes) trained on more than 2,000 everyday videos covering 24 action categories and 120 object categories.
Why human video could train robot hands
The practical payoff is cost. Videos of people doing chores in kitchens, workshops and factories are plentiful, while data recorded from robots is scarce. The filing says that learning from human video reduces the cost of collecting robot-specific data and could improve accuracy.
There is a second use for you. The description also covers a hands-on guide in an extended reality headset: a trainee could ask how a lever on a machine works, and a hologram of a hand would show the motion. This is a patent application, so none of it is a product yet, but it shows where Microsoft's hand-and-object research is pointed.
Microsoft has filed its second application we've tracked on our robot grasping and movement watchlist since November, building on its earlier grip-learning filing.
Source: US 2026/0355959 A1 at the USPTO (PDF). Full patent text and figures are in the official publication.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Microsoft patent news breaks.
Add Patentlyze on Google
One click.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings · 9 sheets
9 drawing sheets from US 2026/0355959 A1 · click any drawing to enlarge
Show the other 1 sheets
More from Microsoft
All 646 Microsoft patents →
More in AI vision
All 339 AI vision patents →
Nvidia
Nvidia Patents a Two-Stage AI Pipeline for Teaching Robots to Grab Objects
Nvidia
Nvidia Files Patent for Indoor Robot Maps That Update Instead of Starting Over
Nvidia
Geometric Fabrics Enable Smarter Robot Grasping
Nvidia
No comments yet