New Patent Teaches Cameras to Track Objects Hidden by Hands

By Patentlyze Team · Updated Jul 21, 2026

When you grip a game controller, your fingers cover half of it, and that's a real problem for cameras trying to track where the controller is pointing. Sony is filing a patent that trains an AI to figure out the hidden parts anyway.

How Sony reads a controller your hand is hiding

Imagine you're playing a VR game and you hold a motion controller in your hand. The camera tracking your controller can only see the parts that stick out from your grip. Your fingers are covering the rest. That gap in visibility can throw off the system trying to calculate exactly where the controller is and which direction it's pointing.

Sony's patent describes a system that learns, in advance, which parts of an object a hand typically covers. It uses that knowledge to train an AI model on images of the object being held. The AI learns to estimate where keypoints (specific reference spots on the object) are, even when they're hidden from view.

The result is a pose-tracking system that can work out an object's full position and orientation in 3D space, even if you're gripping it tightly. That kind of tracking matters a lot in VR and augmented reality, where precise controller position is the whole ballgame.

How the model learns to see through occluded keypoints

The system revolves around pose estimation, which is the technical term for figuring out where an object is in 3D space and which direction it's facing. It does this by finding keypoints, specific, pre-defined spots on the object (think: the tip of a controller, a button location, or a corner edge) and calculating their positions relative to a camera.

The complication the patent addresses is occlusion (when part of the object is blocked from the camera's view, in this case by a human hand). Most standard pose-estimation approaches struggle when keypoints are hidden because the model was never trained to handle the missing data.

Sony's approach works in three steps:

• First, the system identifies which parts of the object a hand would typically cover during normal use.
• Then, it trains a machine learning model using images that include both the object and the hand, specifically targeting the positions of those hidden keypoints.
• Finally, when the trained model sees a new image of a held object, it estimates where all the keypoints are, including ones it can't directly see, and uses those estimates to calculate the object's full 3D pose.

The training data is built around the occlusion problem from the start, rather than treating hidden keypoints as an edge case after the fact.

What this means for PlayStation controllers and AR tracking

For PlayStation VR and similar systems, controller tracking is one of the hardest engineering problems. Hands grip controllers in unpredictable ways, and traditional camera-based tracking can lose accuracy the moment your fingers cover a tracking marker or reference point. A system trained specifically to handle that scenario would produce more reliable, lower-error pose estimates during normal gameplay.

Beyond gaming, the same approach could apply to any situation where a camera tracks a hand-held object: surgical tools, industrial equipment, or augmented reality apps where you pick up and manipulate real-world items. If Sony can generalize this method, it becomes a useful building block for any tracking system that has to deal with human hands getting in the way.

Editorial commentary on a publicly published patent application. Not legal advice.