Sony Patents a System That Turns Your Body Gestures Into In-Game Controls
Sony is working on a way to let your physical movements, captured by a camera, directly control what happens inside a video game, no button presses required.
What Sony's gesture-to-game-control system actually does
You're sitting in front of your TV, playing a game, and instead of tapping buttons on a controller, you wave your hand and a character in the game waves back, or opens a door, or casts a spell. That's the core idea behind this Sony patent.
The system uses a camera to watch you while you play. It spots a gesture you make, figures out what that gesture means, and then uses it to trigger something specific inside the game. The patent calls these in-game objects "assets," which just means anything rendered on screen: a character, an item, an environment piece.
This isn't about replacing controllers entirely. It's about adding a layer where your body becomes part of the input. Sony has been exploring this territory with PlayStation Camera hardware for years, and this patent suggests that work is continuing in software form.
… determining, based at least in part on the image data, a gesture expressed in the image data; and controlling, based at least in part on the gesture, an interaction between assets being rendered in the game.
Translation: The system figures out what physical movement you made and uses it to change the game action.
How the camera reads gestures and maps them to game assets
The patent lays out a three-step process. A camera captures image data while a game is running. Software analyzes that image data to detect and identify a specific gesture (a physical movement made by the player). That identified gesture then triggers a controlled interaction between game assets, meaning it changes something in the rendered game world.
The claims are written at a high level of abstraction, meaning Sony is staking out the broad concept rather than a narrowly specific implementation. The patent does not specify which gestures map to which actions, or what machine-learning model (if any) processes the image feed. That leaves a wide design space inside the claim.
The title of the patent also mentions audio signals and sentiments, which suggests the original research scope may have been broader. The abstract and first claim, however, land squarely on visual gesture recognition from image data. The mismatch between the title and the claim language is worth noting: the filed claims are narrower than the title implies.
- Receives live image data during gameplay
- Identifies a player gesture from that image data
- Maps the gesture to an in-game event or interaction
What this means for how we control games in the future
For players, gesture-based input could make games more physical and expressive, especially for accessibility users who find traditional controllers difficult. A wave, a point, or a tilt of the head as a valid game input opens the door to control schemes that didn't exist before.
For Sony, this connects to its existing investment in camera-based accessories for PlayStation. the pattern in Sony's camera-input filings shows a company that hasn't given up on motion control even after the Kinect era cooled off. Whether this becomes a real feature or stays in the patent portfolio is a separate question, but the filing signals that the problem of "how do players interact with games beyond buttons" is still being worked.
Sony's 516th filing in our Sony coverage since May builds on ideas like shrinking assets for cloud play and real-time accessibility rewrites.
Learning to use a game controller is a real barrier for millions of people, and the cost is not trivial. Families fragment around it, older relatives sit out, and children with limited hand coordination get excluded from something that could otherwise bring them together.
Sony's answer is to let a camera read body movements instead, so the game responds to what your hands and arms do in the air rather than which buttons you press in the right order.
The gap between the patent's title, which promises audio and emotion detection, and the actual legal claim, which covers only visual gesture recognition, makes it hard to know how fully the company intends to solve the problem it has identified. The problem is large enough that a partial solution still matters, but readers should understand this document protects a narrower idea than the headline suggests.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
6 drawing sheets from US 2026/0273406 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →