Sony Patents Technology That Maps a Person's Full Body Shape From a Camera
Sony is working on a system that takes the raw cloud of 3D dots a depth camera captures and automatically maps them onto a jointed skeleton of a human body, assigning each dot a body-part label and a confidence score before locking the skeleton into place.
How Sony's depth-camera body tracker actually works
A security camera stares at an empty hallway all night. The moment a person walks in, it sees a cloud of tiny 3D dots floating in space, but it has no idea which dots are an arm, which are a leg, or how the person is standing. That's the problem Sony is trying to solve.
This patent describes a chip-level system that processes depth sensor output and does two things in sequence. First, it labels every 3D point with a body-part class, such as "left forearm" or "right shoulder," and attaches a score saying how confident it is about that label. Then it takes a pre-defined skeleton model and fits it to those labeled points, the way a tailor drapes a dress form over a mannequin until it matches.
The result is a clean 3D skeleton that represents where a person's body actually is in space, including the angle of each limb. Sony's approach leans on depth sensor hardware rather than a regular camera, so it should work in low-light or high-contrast environments where a standard video feed would struggle.
An information processing device for transforming depth sensor data into 3D skeleton data representing a fitted 3D skeleton model of a body in a scene, comprising circuitry configured to: obtain segmented scene data including a plurality of data points …
Translation: The device turns raw camera depth information into a digital 3D model of a person by processing individual points in space.
How the point cloud gets labeled and fitted to a skeleton
The patent describes an information processing device, essentially a dedicated chip or circuit block, that sits between a depth sensor and whatever application needs body-pose data.
Depth sensors (like the time-of-flight cameras in some gaming consoles and industrial rigs) output a point cloud: a dense set of 3D coordinates, each one representing a surface the sensor pinged. That raw data has no concept of anatomy. The device's first job is segmentation: it takes each point and assigns it a class label (which body part it most likely belongs to) and a class belonging score (a probability between 0 and 1 saying how confident that assignment is). So a point near where a shoulder would typically be might get the label "right shoulder" with a score of 0.87.
Once every point carries that metadata, the device runs a skeleton-fitting step. It takes a known 3D skeleton model (a rigid structure of joints and bones) and mathematically adjusts its joint angles and position until the skeleton aligns as closely as possible with the labeled point cloud. The confidence scores matter here: points with high scores pull the skeleton toward them more strongly than uncertain ones.
- Input: raw 3D point cloud from a depth sensor
- Step 1: label each point with a body-part class and a confidence score
- Step 2: fit a pre-built skeleton model to those labeled, scored points
- Output: a clean 3D skeleton representing the person's pose
… each data point includes a 3D position, a class label indicating one of a plurality of classes, each class representing a body part, and a class belonging score indicating a likeliness that the data point belongs to the class indicated by the class label …
Translation: The system labels every point in the camera view as a specific body part and calculates how certain it is about that choice.
What this means for cameras that track human movement
Body-pose tracking from depth cameras has been around since the original Xbox Kinect era, but doing it accurately and efficiently enough to run on embedded hardware, without offloading heavy computation to a server, remains a real engineering challenge. Sony Semiconductor Solutions makes image sensors and depth camera chips, so a patent like this points toward baking skeleton-tracking directly into sensor hardware, which would let cameras deliver ready-to-use pose data rather than raw frames that need separate processing.
That matters for anything from sports analysis systems and retail foot-traffic tools to healthcare motion capture, where a self-contained camera that hands you a skeleton instead of a video feed is far easier to deploy. Sony's depth-sensor and body-tracking interesting tech patents sit alongside a wider cluster of filings from camera and chip makers racing to move AI inference closer to the sensor itself, rather than in the cloud.
The patent combines two well-understood steps, point-cloud segmentation and skeleton fitting, into a single circuit-level device, and the applicant is Sony Semiconductor Solutions, meaning this is headed for silicon. Freezing that logic into hardware at tape-out costs far more time and money than shipping a software update, so Sony evidently sees a product category where on-chip inference pays for itself, likely an embedded depth camera module. The plausible end state is a sensor module that exposes a skeleton data stream over a standard interface, dropping into existing systems the way a microphone exposes audio.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
4 drawing sheets from US 2026/0237084 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →